CN112949424B - Neuromorphic visual sampling method and device - Google Patents
Neuromorphic visual sampling method and device Download PDFInfo
- Publication number
- CN112949424B CN112949424B CN202110168780.0A CN202110168780A CN112949424B CN 112949424 B CN112949424 B CN 112949424B CN 202110168780 A CN202110168780 A CN 202110168780A CN 112949424 B CN112949424 B CN 112949424B
- Authority
- CN
- China
- Prior art keywords
- signal
- pulse
- sampling
- brightness
- coding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000005070 sampling Methods 0.000 title claims abstract description 87
- 230000000007 visual effect Effects 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013528 artificial neural network Methods 0.000 claims abstract description 28
- 230000000306 recurrent effect Effects 0.000 claims description 19
- 238000013527 convolutional neural network Methods 0.000 claims description 18
- 230000004927 fusion Effects 0.000 claims description 15
- 238000009825 accumulation Methods 0.000 claims description 10
- 230000003287 optical effect Effects 0.000 claims description 10
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 5
- 230000003044 adaptive effect Effects 0.000 claims description 4
- 125000004122 cyclic group Chemical group 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 230000001537 neural effect Effects 0.000 abstract description 6
- 238000004364 calculation method Methods 0.000 abstract description 5
- 230000008447 perception Effects 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 239000000284 extract Substances 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 2
- 210000005036 nerve Anatomy 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a neural morphology vision sampling method and a device, wherein the method comprises the following steps: collecting light signals of different position points in a current scene and converting the light signals into brightness signals; inputting the converted brightness signal into a trained pulse sampling model, and sampling and coding the input brightness signal by the pulse sampling model to obtain a pulse array signal; and inputting the pulse array signal into a trained visual task model, and carrying out visual analysis on the pulse array signal by the visual task model to obtain an analysis result. According to the invention, a pulse array signal is acquired without using a dynamic vision sensor or an integral vision sensor, the neural form vision sampling is carried out through a neural network, and the vision analysis calculation task and the sampling are integrated, so that various vision sampling models can be flexibly customized to adapt to the application requirements of multiple scenes and multiple tasks, and the self-use perception performance is improved.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a neuromorphic vision sampling method and device.
Background
The neuromorphic vision sensor has the advantages of high time domain resolution, high dynamic range, low data redundancy, low power consumption and the like, and has wide application prospects in the fields of automatic driving, unmanned aerial vehicle vision navigation and the like. Currently, the common neuromorphic Vision sensors include DVS (dynamic Vision sensor) and IVS (integrated Vision sensor).
In the existing methods of using a neuromorphic visual sensor to perform various visual tasks (such as identification, reconstruction and classification), one method is to convert a pulse array signal output by the neuromorphic visual sensor into a frame of picture and apply a traditional visual algorithm to analyze the picture on the basis of the frame of picture; and the other method is to extract the space-time characteristics of the pulse array signals through a depth network and perform subsequent visual tasks.
No matter which method is adopted, the scene needs to be shot through the nerve morphology vision sensor, then the network is designed and trained according to the existing nerve morphology representation, and the sampling model is fixed, so that the method cannot adapt to the application requirements of multiple scenes and multiple tasks.
Disclosure of Invention
The present invention provides a neuromorphic visual sampling method and apparatus for addressing the above-mentioned deficiencies of the prior art, and the objective is achieved by the following technical solution.
A first aspect of the present invention provides a neuromorphic visual sampling method, the method comprising:
collecting light signals of different position points in a current scene and converting the light signals into brightness signals;
inputting the converted brightness signal into a trained pulse sampling model, and sampling and coding the input brightness signal by the pulse sampling model to obtain a pulse array signal;
and inputting the pulse array signal into a trained visual task model, and carrying out visual analysis on the pulse array signal by the visual task model to obtain an analysis result.
A second aspect of the invention provides a neuromorphic visual sampling device, the device comprising:
the photoelectric conversion module is used for collecting optical signals of different position points in the current scene and converting the optical signals into brightness signals;
the pulse sampling coding module is used for inputting the brightness signal converted by the photoelectric conversion module into a trained pulse sampling model, and sampling and coding the input brightness signal by the pulse sampling model to obtain a pulse array signal;
and the visual task module is used for inputting the pulse array signal obtained by the pulse sampling coding module into a trained visual task model so as to perform visual analysis on the pulse array signal by the visual task model to obtain an analysis result.
Based on the neuromorphic visual sampling method and the neuromorphic visual sampling device in the first aspect and the second aspect, the neuromorphic visual sampling method and the neuromorphic visual sampling device have the following beneficial effects:
according to the invention, a pulse array signal is acquired without using a dynamic vision sensor or an integral vision sensor, the neural form vision sampling is carried out through a neural network, and the vision analysis calculation task and the sampling are integrated, so that various vision sampling models can be flexibly customized to adapt to the application requirements of multiple scenes and multiple tasks, and the self-use perception performance is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a flow chart illustrating an embodiment of a neuromorphic visual sampling method according to an exemplary embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a pulse sampling model according to an exemplary embodiment of the present invention;
FIG. 3 is a schematic diagram of an integrated pulse array signal according to the present invention;
FIG. 4 is a schematic diagram of a differential pulse array signal according to the present invention;
FIG. 5 is a diagram illustrating a visual task model in a target recognition scenario, in accordance with an exemplary embodiment of the present invention;
FIG. 6 is a diagram illustrating a configuration of a visual task model in an image reconstruction scene in accordance with an exemplary embodiment of the present invention;
fig. 7 is a schematic structural diagram of a neuromorphic visual sampling device according to an exemplary embodiment of the present invention.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. The word "if," as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination," depending on the context.
In the existing neural morphology vision sensor, the dynamic vision sensor is also called an event camera, compared with the traditional camera, the dynamic vision sensor does not use a sampling mode of a fixed frame frequency, but asynchronously encodes local intensity contrast into a time stamp event, the time resolution of the time stamp event is 10M events/s, and a high-speed motion event can be recorded into a space-time sparse event lattice, namely a differential pulse array, so that the dynamic vision sensor is suitable for application of a dynamic object scene and is not suitable for a scene with less motion. The integral vision sensor is an integral pulse array signal obtained by integral coding of local intensity and synchronous distribution through setting a frame rate, the time resolution of the integral pulse array signal can reach 40KHz at most, high-speed motion can be coded into a pulse three-dimensional space-time array comprising 0 and 1, and the integral vision sensor is suitable for application of static object scenes and is not suitable for scenes with much motion.
Therefore, the two types of vision sensors are applicable to single scene and fixed sampling models, and cannot adapt to the application requirements of multiple scenes and multiple tasks.
In order to solve the technical problems, the invention does not use a dynamic vision sensor or an integral vision sensor to obtain a pulse array signal, but provides a neural form vision sampling scheme defined by a neural network, and integrates a vision analysis calculation task and sampling, so that various vision sampling models can be flexibly customized to adapt to the application requirements of multiple scenes and multiple tasks, and the self-use perception performance is improved.
The neuromorphic visual sampling scheme proposed by the present invention is described in detail below with specific examples.
Fig. 1 is a flowchart illustrating an embodiment of a neuromorphic visual sampling method applied to a sampling chip according to an exemplary embodiment of the present invention, as shown in fig. 1, the neuromorphic visual sampling method including the following steps:
step 101: light signals of different position points in the current scene are collected and converted into brightness signals.
In some embodiments, the luminance signal is obtained by acquiring light signals of different position points in the current scene in real time, converting the acquired light signals into electric signals, and performing logarithmic coding on the electric signals.
The light signals of all position points collected at the same moment are subjected to photoelectric conversion to obtain a brightness signal, and therefore the brightness signal obtained by converting the current scene is a brightness signal which is continuous in time and has a spatial neighborhood relationship. That is, for the same time, the luminance signals of different position points may form an orderly arranged luminance matrix according to the spatial neighborhood light contexts of the position points.
Step 102: and inputting the converted brightness signal into a trained pulse sampling model, and sampling and coding the input brightness signal by the pulse sampling model to obtain a pulse array signal.
In some embodiments, referring to the detailed structure of the pulse sampling model shown in fig. 2, the luminance signal is denoised and luminance-adaptive processed by the convolutional neural network in the pulse sampling model to obtain a processed luminance signal, and the processed luminance signal is output to the cyclic neural network in the pulse sampling model, and then the processed luminance signal is sampled and encoded by the cyclic neural network to obtain a pulse array signal.
Further, the specific processing flow for the convolutional neural network includes: the method comprises the steps of extracting features of an array formed by luminance signals of different position points at the same time through a feature extraction module in a convolutional neural network, outputting the extracted feature array to a denoising module and a luminance self-adaption module in the convolutional neural network, denoising the extracted feature array through the denoising module to obtain a denoised array, outputting the denoised array to a fusion module, conducting luminance self-adaption processing on the extracted features through the luminance self-adaption module to obtain a self-adaption processed array, outputting the self-adaption processed array to the fusion module, and finally fusing the array formed by the luminance signals of different position points at the same time, the denoised array and the self-adaption processed array through the fusion module to obtain a processed luminance signal.
The method comprises the steps of obtaining luminance signals of different positions at the same time, inputting the luminance signals of different positions at the same time by a convolutional neural network, obtaining a self-adaptive processed array by a convolutional neural network, obtaining a denoised array by a convolutional neural network, and obtaining a fusion formula of a fusion module by a convolutional neural network.
Further, the specific processing flow of the recurrent neural network includes that the processed brightness signal is differentially encoded through an integral encoding network in the recurrent neural network to obtain an integral pulse array signal, and/or the processed brightness signal is differentially encoded through a differential encoding network in the recurrent neural network to obtain a differential pulse array signal.
The processing flow for the integral coding network comprises the following steps: and integrating the integral information recorded by the hidden layer, the integral information of the processed brightness signal on time and a set threshold value through a feature integration module in the integral coding network to generate an integral pulse array signal.
The output formula of the integral coding network is o = sigma (i + h-t), wherein i, h and t respectively represent input luminance signals, integral information and a set threshold value recorded by the hidden layer are recorded, the set threshold value t is preset or obtained through training, sigma is a sigmoid function, and the calculation principle of sigma is that the integral information recorded by the hidden layer is differed with the set threshold value in time to obtain output. The updating formula of the hidden layer is h '= h-o × t, h' is the hidden layer updated after output, and h is the integral information recorded by the original hidden layer.
As shown in fig. 3, a binary matrix representation is assigned to each time instant for generating a pulse array signal based on luminance integration and distribution.
It should be noted that in different visual task scenarios, the output of the integral coding network can be changed by changing the value of the set threshold t in the integral coding network, and if the integral pulse array signal needs to be output in the visual task scenario, the set threshold is set to a proper value, and if the integral pulse array signal is not needed in the visual task scenario, the set threshold can be set to a large value, so that the output is 0.
The processing flow of the differential coding network comprises the following steps: and fusing the brightness information recorded by the hidden layer, the change information of the processed brightness signal in time and a set threshold value through a characteristic fusion module in the differential coding network to generate a differential pulse array signal.
The output formula of the differential coding network is o = sigma (i-h-t) -sigma (h-i-t), wherein i represents an input brightness signal, h identifies brightness information recorded by the hidden layer, t is a set threshold value, and sigma is a sigmoid function. The hidden layer updating formula is h '= o × i + (1-o × o) × h, h' is the updated hidden layer after output, and h is the brightness information recorded by the original hidden layer.
As shown in fig. 4, the generated pulse array signal issued based on brightness change is characterized by event quadruple coding, and quadruple information comprises abscissa and ordinate, time and polarity. Wherein the abscissa and the ordinate can be mapped to a position point in the scene, and the polarity represents the brightness change condition.
It should be noted that, in different visual task scenarios, the output of the differential coding network can be changed by changing the value of the set threshold t in the differential coding network, if the differential pulse array signal needs to be output in the visual task scenario, the set threshold is set to a proper value, and if the differential pulse array signal is not needed in the visual task scenario, the set threshold can be set to a large value, so that the output is 0.
Therefore, the type of the pulse array signal output by the pulse sampling model can be changed by flexibly setting different network parameters.
Step 103: and inputting the pulse array signal into a trained visual task model, and carrying out visual analysis on the pulse array signal by the visual task model to obtain an analysis result.
In an embodiment, when the visual task model is used for target identification, a pulse accumulation plane and a receptive field feature vector can be respectively generated by the visual task model according to the time dimension feature and the space dimension feature of the differential pulse array signal, and a target identification result is obtained according to the pulse accumulation plane and the receptive field feature vector identification.
Referring to fig. 5, in a target identification scene, a visual task model extracts spatial dimension features of a differential pulse array signal, reconstructs the spatial dimension features to generate a pulse accumulation plane, extracts time dimension features of the differential pulse array signal, extracts the time dimension features to generate a receptive field feature vector, and then a target identification module performs target identification according to the pulse accumulation plane and the receptive field feature vector to obtain an identification result.
The pulse accumulation plane refers to a frame of image represented by gray scale values, and the gray scale value of each pixel in the image is calculated according to the polarity value of a position point in a scene corresponding to the pixel position in a period of time to obtain a gray scale value.
In another embodiment, when the visual task model is used for image reconstruction, a receptor field characteristic vector is generated by the visual task model according to the time dimension characteristic of the differential pulse array signal, and image reconstruction is performed according to the integral pulse array signal and the receptor field characteristic vector to obtain a reconstructed image.
Referring to fig. 6, in an image reconstruction scene, the visual task model extracts time dimension features of the differential pulse array signal, extracts the time dimension features to generate a receptive field feature vector, and then the reconstruction module performs image reconstruction according to the pulse accumulation plane and the integral pulse array signal.
The system comprises a receiving field characteristic vector, an integral pulse array signal and a reconstruction image, wherein the receiving field characteristic vector is used for reconstructing the contour and the position of an object, the integral pulse array signal is used for reconstructing the detail texture of the object, and the reconstructed image of the current scene is obtained by combining the reconstructed contour and the position of the object and the detail texture of the object.
So far, the flow shown in fig. 1 is completed, and in the invention, a pulse array signal is acquired without using a dynamic vision sensor or an integral vision sensor, and neural form vision sampling is performed through a neural network, and a vision analysis calculation task and sampling are integrated, so that various vision sampling models can be flexibly customized to adapt to multi-scene and multi-task application requirements, and the self-use perception performance is improved.
Fig. 7 is a flowchart illustrating an embodiment of a neuromorphic visual sampling device applied to a sampling chip according to an exemplary embodiment of the present invention, as shown in fig. 7, the neuromorphic visual sampling device including:
the photoelectric conversion module 710 is configured to collect optical signals at different position points in a current scene and convert the optical signals into luminance signals;
the pulse sampling coding module 720 is configured to input the luminance signal obtained by conversion by the photoelectric conversion module into a trained pulse sampling model, so that the input luminance signal is sampled and coded by the pulse sampling model to obtain a pulse array signal;
and the visual task module 730 is configured to input the pulse array signal obtained by the pulse sampling coding module into a trained visual task model, so that the visual task model performs visual analysis on the pulse array signal to obtain an analysis result.
In an optional implementation manner, the photoelectric conversion module 710 is specifically configured to acquire optical signals of different position points in a current scene in real time; converting the collected optical signals into electric signals, and carrying out logarithmic coding on the electric signals to obtain brightness signals; the luminance signal obtained by converting the current scene is a luminance signal which is continuous in time and has a spatial neighborhood relationship.
In an optional implementation manner, the pulse sampling coding module 720 is specifically configured to, in a process of obtaining a pulse array signal by sampling and coding an input luminance signal through the pulse sampling model, perform denoising processing and luminance adaptive processing on the luminance signal through a convolutional neural network in the pulse sampling model to obtain a processed luminance signal, and output the processed luminance signal to a cyclic neural network in the pulse sampling model; and sampling and coding the processed brightness signal through the recurrent neural network to obtain a pulse array signal.
In an optional implementation manner, the pulse sampling coding module 720 is specifically configured to, in a process of performing denoising processing and luminance adaptive processing on the luminance signal through a convolutional neural network in the pulse sampling model to obtain a processed luminance signal, perform feature extraction on an array formed by luminance signals at different positions at the same time through a feature extraction module in the convolutional neural network, and output the array to a denoising module and a luminance adaptive module in the convolutional neural network; denoising the extracted characteristic array through the denoising module to obtain a denoised array and output the denoised array to a fusion module in the convolutional neural network; performing brightness self-adaptive processing on the extracted features through the brightness self-adaptive module to obtain a self-adaptively processed array and outputting the array to the fusion module; and fusing an array consisting of brightness signals of different positions at the same time, the denoised array and the adaptively processed array through the fusion module to obtain the processed brightness signals.
In an optional implementation manner, the pulse sampling and encoding module 720 is specifically configured to perform differential encoding on the processed luminance signal through an integral encoding network in the recurrent neural network to obtain an integral pulse array signal in a process of performing sampling encoding on the processed luminance signal through the recurrent neural network to obtain a pulse array signal; and/or carrying out differential coding on the processed brightness signal through a differential coding network in the recurrent neural network to obtain a differential pulse array signal.
In an optional implementation manner, the pulse sampling encoding module 720 is specifically configured to, in the process of obtaining an integral pulse array signal by differentially encoding the processed luminance signal through an integral encoding network in the recurrent neural network, fuse, by a feature fusion module in the integral encoding network, integral information recorded by the hidden layer, integral information of the processed luminance signal over time, and a set threshold, so as to generate the integral pulse array signal.
In an optional implementation manner, the pulse sampling encoding module 720 is specifically configured to, in the process of obtaining a differential pulse array signal by differentially encoding the processed luminance signal through a differential encoding network in the recurrent neural network, fuse, by a feature fusion module in the differential encoding network, luminance information recorded by the hidden layer, change information of the processed luminance signal in time, and a set threshold, so as to generate the differential pulse array signal.
In an optional implementation manner, the visual task module 730 is specifically configured to perform visual analysis on the pulse array signal through a visual task model to obtain an analysis result, and when the visual task model is used for target identification, respectively generate a pulse accumulation plane and a receptive field feature vector according to a time dimension feature and a space dimension feature of a differential pulse array signal through the visual task model, and identify according to the pulse accumulation plane and the receptive field feature vector to obtain a target identification result.
In an optional implementation manner, the visual task module 730 is specifically configured to perform visual analysis on the pulse array signal through a visual task model to obtain an analysis result, and when the visual task model is used for image reconstruction, generate a receptive field feature vector according to a time dimension feature of the differential pulse array signal through the visual task model, and perform image reconstruction according to the integral pulse array signal and the receptive field feature vector to obtain a reconstructed image.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the invention. One of ordinary skill in the art can understand and implement it without inventive effort.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (9)
1. A neuromorphic visual sampling method applied to a visual sensor, comprising:
collecting light signals of different position points in a current scene and converting the light signals into brightness signals;
inputting the converted brightness signal into a trained pulse sampling model, and sampling and coding the input brightness signal by using the pulse sampling model to obtain a pulse array signal, wherein the processed brightness signal is sampled and coded by using a recurrent neural network to obtain the pulse array signal;
inputting the pulse array signal into a trained visual task model, and carrying out visual analysis on the pulse array signal by the visual task model to obtain an analysis result;
the method for obtaining the pulse array signal by sampling and coding the processed brightness signal through the recurrent neural network comprises the following steps:
carrying out integral coding on the processed brightness signal through an integral coding network in the recurrent neural network to obtain an integral pulse array signal; and/or the presence of a gas in the gas,
and carrying out differential coding on the processed brightness signal through a differential coding network in the recurrent neural network to obtain a differential pulse array signal.
2. The method of claim 1, wherein the acquiring and converting the light signals of different position points in the current scene into the luminance signal comprises:
acquiring optical signals of different position points in a current scene in real time;
converting the collected optical signals into electric signals, and carrying out logarithmic coding on the electric signals to obtain brightness signals;
the luminance signal obtained by converting the current scene is a luminance signal which is continuous in time and has a spatial neighborhood relationship.
3. The method of claim 1, wherein the pulse sampling model sample-codes the input luminance signal to obtain a pulse array signal, and comprises:
denoising and brightness self-adaptive processing are carried out on the brightness signal through a convolutional neural network in the pulse sampling model to obtain a processed brightness signal, and the processed brightness signal is output to a cyclic neural network in the pulse sampling model;
and sampling and coding the processed brightness signal through the recurrent neural network to obtain a pulse array signal.
4. The method according to claim 3, wherein the denoising and luminance adaptive processing are performed on the luminance signal through a convolutional neural network in the present pulse sampling model to obtain a processed luminance signal, and the method comprises:
the characteristic extraction module in the convolutional neural network is used for extracting the characteristics of an array formed by brightness signals of different position points at the same time and outputting the extracted characteristics to a denoising module and a brightness self-adaptive module in the convolutional neural network;
denoising the extracted characteristic array through the denoising module to obtain a denoised array and output the denoised array to a fusion module in the convolutional neural network;
performing brightness self-adaptive processing on the extracted features through the brightness self-adaptive module to obtain a self-adaptively processed array and outputting the array to the fusion module;
and fusing an array consisting of brightness signals of different positions at the same time, the denoised array and the adaptively processed array through the fusion module to obtain the processed brightness signals.
5. The method of claim 1, wherein the differentially encoding the processed luminance signal by an integral encoding network in the recurrent neural network to obtain an integral pulse array signal comprises:
and integrating the integral information recorded by the hidden layer, the integral information of the processed brightness signal on time and a set threshold value through a feature integration module in the integral coding network to generate an integral pulse array signal.
6. The method according to claim 1, wherein the differentially encoding the processed luminance signal by a differential encoding network in the recurrent neural network to obtain a differential type pulse array signal comprises:
and fusing the brightness information recorded by the hidden layer, the change information of the processed brightness signal in time and a set threshold value through a feature fusion module in the differential coding network to generate a differential pulse array signal.
7. The method of claim 1, wherein the visual task model performs a visual analysis on the pulse array signal to obtain an analysis result, comprising:
when the visual task model is used for target identification, a pulse accumulation plane and a receptive field characteristic vector are respectively generated through the visual task model according to the time dimension characteristic and the space dimension characteristic of the differential pulse array signal, and a target identification result is obtained through identification according to the pulse accumulation plane and the receptive field characteristic vector.
8. The method of claim 1, wherein the visual task model performs a visual analysis on the pulse array signal to obtain an analysis result, comprising:
when the visual task model is used for image reconstruction, a reception field characteristic vector is generated through the visual task model according to the time dimension characteristic of the difference type pulse array signal, and image reconstruction is carried out according to the integral type pulse array signal and the reception field characteristic vector to obtain a reconstructed image.
9. A neuromorphic visual sampling device, for application to a visual sensor, comprising:
the photoelectric conversion module is used for collecting optical signals of different position points in the current scene and converting the optical signals into brightness signals;
the pulse sampling coding module is used for inputting the brightness signal converted by the photoelectric conversion module into a trained pulse sampling model, and sampling and coding the input brightness signal by using the pulse sampling model to obtain a pulse array signal, wherein the processed brightness signal is sampled and coded by using a recurrent neural network to obtain the pulse array signal;
the visual task module is used for inputting the pulse array signal obtained by the pulse sampling coding module into a trained visual task model so as to perform visual analysis on the pulse array signal by the visual task model to obtain an analysis result;
the method for obtaining the pulse array signal by sampling and coding the processed brightness signal through the recurrent neural network comprises the following steps:
carrying out integral coding on the processed brightness signal through an integral coding network in the circulating neural network to obtain an integral pulse array signal; and/or the presence of a gas in the gas,
and carrying out differential coding on the processed brightness signal through a differential coding network in the recurrent neural network to obtain a differential pulse array signal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110168780.0A CN112949424B (en) | 2021-02-07 | 2021-02-07 | Neuromorphic visual sampling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110168780.0A CN112949424B (en) | 2021-02-07 | 2021-02-07 | Neuromorphic visual sampling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112949424A CN112949424A (en) | 2021-06-11 |
| CN112949424B true CN112949424B (en) | 2023-02-07 |
Family
ID=76243295
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110168780.0A Active CN112949424B (en) | 2021-02-07 | 2021-02-07 | Neuromorphic visual sampling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112949424B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114581490B (en) * | 2022-04-29 | 2022-09-20 | 北京大学 | Scene setting method and device, control equipment and readable storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107333040B (en) * | 2017-07-13 | 2020-02-21 | 中国科学院半导体研究所 | Bionic visual imaging and processing device |
| CN111950720A (en) * | 2020-08-26 | 2020-11-17 | 南京大学 | Novel brain-like vision system |
-
2021
- 2021-02-07 CN CN202110168780.0A patent/CN112949424B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112949424A (en) | 2021-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110119703B (en) | Human body action recognition method fusing attention mechanism and spatio-temporal graph convolutional neural network in security scene | |
| CN108921799B (en) | Remote sensing image thin cloud removing method based on multi-scale collaborative learning convolutional neural network | |
| CN111091045A (en) | Sign language identification method based on space-time attention mechanism | |
| CN110120049B (en) | Method for jointly estimating scene depth and semantics by single image | |
| Anvari et al. | Dehaze-GLCGAN: unpaired single image de-hazing via adversarial training | |
| CN112686928A (en) | Moving target visual tracking method based on multi-source information fusion | |
| CN113112521B (en) | Motion detection method based on pulse array | |
| CN116311254B (en) | Image target detection method, system and equipment under severe weather condition | |
| CN114245007A (en) | High frame rate video synthesis method, device, equipment and storage medium | |
| CN111798370A (en) | Manifold constraint-based event camera image reconstruction method and system | |
| CN115484410A (en) | Event camera video reconstruction method based on deep learning | |
| CN112949424B (en) | Neuromorphic visual sampling method and device | |
| Jiang et al. | Event-based low-illumination image enhancement | |
| CN114119356A (en) | Method for converting thermal infrared image into visible light color image based on cycleGAN | |
| CN113901931B (en) | Behavior recognition method of infrared and visible light video based on knowledge distillation model | |
| CN113014805B (en) | Combined sampling method and device for simulating fovea and periphery of retina | |
| CN113628143A (en) | Weighted fusion image defogging method and device based on multi-scale convolution | |
| CN111626944A (en) | Video deblurring method based on space-time pyramid network and natural prior resistance | |
| CN114821239B (en) | Method for detecting plant diseases and insect pests in foggy environment | |
| CN116385283A (en) | Image deblurring method and system based on event camera | |
| CN113191991B (en) | Information bottleneck-based multi-mode image fusion method, system, equipment and medium | |
| Liu et al. | Motion-oriented hybrid spiking neural networks for event-based motion deblurring | |
| CN113255514B (en) | Behavior identification method based on local scene perception graph convolutional network | |
| CN116091337A (en) | Image enhancement method and device based on event signal nerve coding mode | |
| CN114998402A (en) | Monocular depth estimation method and device for pulse camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |