CN110555523B - Short-range tracking method and system based on impulse neural network - Google Patents

Short-range tracking method and system based on impulse neural network Download PDF

Info

Publication number
CN110555523B
CN110555523B CN201910668450.0A CN201910668450A CN110555523B CN 110555523 B CN110555523 B CN 110555523B CN 201910668450 A CN201910668450 A CN 201910668450A CN 110555523 B CN110555523 B CN 110555523B
Authority
CN
China
Prior art keywords
neural network
pulse
layer
convolutional
impulse
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910668450.0A
Other languages
Chinese (zh)
Other versions
CN110555523A (en
Inventor
洪苑乾
李金生
肖菲
晏新亮
吴善农
柳博予
李欣
董蓓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Third Bureau Intelligent Technology Co Ltd
Original Assignee
China Construction Third Bureau Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Third Bureau Intelligent Technology Co Ltd filed Critical China Construction Third Bureau Intelligent Technology Co Ltd
Priority to CN201910668450.0A priority Critical patent/CN110555523B/en
Publication of CN110555523A publication Critical patent/CN110555523A/en
Priority to PCT/CN2020/089907 priority patent/WO2021012752A1/en
Application granted granted Critical
Publication of CN110555523B publication Critical patent/CN110555523B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/061—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using biological neurons, e.g. biological neurons connected to an integrated circuit
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Neurology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a short-range tracking method and a short-range tracking system based on a pulse neural network, which relate to the technical field of artificial intelligence, and the method comprises the steps of carrying out pulse coding on an input image based on an attention mechanism; modifying the structure of the convolutional neural network to transfer the parameters of the convolutional neural network into the impulse neural network and reconstruct the impulse neural network; calculating pulse similarity between corresponding feature points in adjacent image frames of the input image to obtain regional similarity; and tracking the target in the input image by using the reconstructed impulse neural network. The reconstructed impulse neural network effectively combines the strong feature extraction characteristic of the convolutional neural network and the high-efficiency calculation characteristic of the impulse neural network.

Description

Short-range tracking method and system based on impulse neural network
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a short-range tracking method and a short-range tracking system based on a pulse neural network.
Background
With the development and wide application of artificial intelligence related technologies, brain-like computing is more and more concerned by researchers, although the structure of a traditional deep neural network is inspired by the brain, the structure is fundamentally different from the brain in the aspects of computing and learning rules, information is transmitted by taking a pulse sequence as a carrier in the brain, and the pulse neural network with high bionics is generated in order to simulate the information processing mechanism of the biological neuron.
The impulse neural network is called as a third generation neural network, has been the focus of research in pattern recognition problems such as image classification, belongs to the leading edge technical research topic in the field of artificial intelligence, has the advantages of high calculation efficiency, low energy consumption, less occupied resources, easy hardware realization and the like, is an ideal choice for researching brain neural calculation and coding strategies, has important significance for promoting the development of the artificial neural network through the theory and application research of the impulse neural network, and can also promote the research of edge devices such as a novel artificial intelligence chip of a non-von Neumann computing architecture and the like.
At present, some preliminary achievements have been made on the research of the impulse neural network, but the application of the impulse neural network is still in a starting stage, the impulse neural network is mainly used for the aspects of handwritten number recognition, image segmentation and the like, and is difficult to apply to a complex visual scene, and the key of the problem is that the neuron function in the impulse neural network is not microminiature, the training cannot be performed by using the traditional error back propagation mode, and the training algorithm with low efficiency at present cannot overcome the training problem of the complex impulse neural network model, so that the bottleneck is brought to the popularization and application of the impulse neural network.
On the other hand, tracking is an important research direction in the field of computer vision, and has specific applications in many fields such as automatic driving, safety, behavior recognition, human-computer interaction and the like. In recent years, deep learning models based on convolutional neural networks, automatic encoders and the like have a lot of progress on tracking technology, and the deep learning models have remarkable feature extraction capability, and the deep learning models cannot be applied to edge equipment due to large calculation amount, large occupied resources and need to be accelerated depending on a top level display card, but if the deep learning models can be fused with the characteristics of high calculation efficiency, easiness in hardware implementation and the like of the impulse neural network model, the application is possible, but at present, the impulse neural network model is mostly used for classification, specific processing of output impulses is not needed, and the field of tracking and the like which needs additional operation after output is not tried yet.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a short-range tracking method and a short-range tracking system based on a pulse neural network, wherein the reconstructed pulse neural network effectively combines the strong feature extraction characteristic of a convolutional neural network and the high-efficiency calculation characteristic of the pulse neural network.
The invention provides a short-range tracking method based on a pulse neural network, which comprises the following steps:
pulse coding the input image based on an attention mechanism;
modifying the structure of the convolutional neural network to transfer the parameters of the convolutional neural network into the impulse neural network and reconstruct the impulse neural network;
calculating pulse similarity between corresponding feature points in adjacent image frames of the input image to obtain regional similarity;
and tracking the target in the input image by using the reconstructed impulse neural network.
On the basis of the technical scheme, the pulse coding is performed on the input image based on the attention mechanism, and the specific steps include:
carrying out convolution operation on the input image by using a 3 x 3 receptive field region operator to obtain a characteristic diagram;
based on the sequence of the characteristic values from big to small, sorting the pixel points in the characteristic graph, taking a preset number of pixel points according to the sorting, and setting the characteristic values of the taken pixel points as the characteristic values of the first ranked pixel points;
calculating the pulse distribution number s of each pixel point in the characteristic diagrami，jThe calculation formula is as follows:
Figure BDA0002140873410000031
wherein p ismaxIs the maximum pixel value, p, of a pixel point in the feature mapminIs the minimum pixel value, p, of a pixel point in the feature mapi,jThe gray value of a pixel point in the feature map is obtained, and S is the number of pulses of the feature map;
calculating the frequency f of each pixel point in the characteristic diagrami,jAnd based on the calculated pulse distribution number s of each pixel pointi，jGenerating a pulse code sequence, and calculating the frequency of each pixel point in the characteristic diagram by the following calculation formula:
fi,j＝T/si,j
wherein T is the total pulse time of the characteristic diagram.
On the basis of the above technical solution, the structure of the convolutional neural network is modified, and the specific modification process of the convolutional neural network structure is as follows:
for an input layer, carrying out normalization processing on an input image;
for the convolutional layer, all offsets in the convolutional layer are set to be 0, and the sizes and initialization settings of other original cores are unchanged;
for an activation layer, the position of an activation function is originally required to be used, and the activation function is replaced by a relu () activation function;
for the Pooling layer, if the network uses the neurons with single pulse output, the original Max-Pooling layer or Average-Pooling layer is kept in the Pooling layer, and if the network uses the neurons with multi-pulse output, the Max-Pooling layer of the Pooling layer is modified into the Average-Pooling layer;
setting all offsets in the full connection layer to be 0, keeping the original neuron number and initialization of the full connection layer unchanged, and using an L2 regularization strategy for the weight of the full connection layer in a training stage; and
the erasure cannot directly represent the layer and sets the type of all weights in the convolutional neural network to a 16-bit floating point type.
On the basis of the technical scheme, the convolutional neural network parameters are migrated into the impulse neural network, the impulse neural network is reconstructed, and the specific process for constructing the impulse neural network structure is as follows:
for the convolutional layer, constructing convolutional kernels with the same number and the same size as those of the convolutional layer of the convolutional neural network, and then directly transferring weight parameters of the convolutional neural network to construct the convolutional layer of the impulse neural network;
for the Pooling layer, if the network uses a single-pulse output neuron, the Max-Powing layer of the convolutional neural network corresponds to the earliest pulse emitting time in a 2 x 2 area input by the Pooling layer of the impulse neural network, and the Average-Powing layer of the convolutional neural network corresponds to the Average pulse time of the Pooling layer of the impulse neural network; if the network uses a multi-pulse output neuron, calculating the Average-Pooling layer of the Pooling layer in a convolution mode;
for the active layer, the active layer of the migration convolution neural network forms the active layer of the pulse neural network, the linear activation mode in the pulse neural network is used for calculating the accumulated voltage for the position using the relu () activation function in the migrated active layer, when the accumulated voltage reaches the release threshold, the release generates the output pulse, the membrane voltage is reset to the rest potential, when the accumulated voltage is smaller than the release threshold, the current voltage value is recorded, and when the accumulated voltage is lower than the rest potential, the membrane voltage is reset to the rest potential;
and for the full connection layer, constructing the neurons with the same number as the full connection layer of the convolutional neural network, and directly transferring the weight of the full connection layer of the convolutional neural network to form the full connection layer of the impulse neural network.
On the basis of the above technical solution, in the calculating the pulse similarity between corresponding feature points in adjacent image frames of the input image to obtain the region similarity, the calculation process of the pulse similarity between two feature points is as follows:
calculating the distance between the current time t and the previous pulse emitting time in the pulse code sequence
Figure BDA0002140873410000051
The calculation formula is as follows:
Figure BDA0002140873410000052
wherein,
Figure BDA0002140873410000053
the next pulse in the nth pulse code sequence is issued for the current time t;
calculating the distance delta t between the current time t and the next pulse sending time in the pulse code sequenceP(t) the calculation formula is:
Figure BDA0002140873410000054
wherein,
Figure BDA0002140873410000055
for the time of issuance of the previous pulse in pulse code sequence 1 for the current instant t,
Figure BDA0002140873410000056
the sending time of the previous pulse in the pulse code sequence 2 is the current time t;
calculating a pulse emission time difference delta t after the current time t of two pulse code sequencesF(t) the calculation formula is:
Figure BDA0002140873410000057
wherein,
Figure BDA0002140873410000058
for the firing time of the following pulse in the pulse code sequence 1 at the current instant t,
Figure BDA0002140873410000059
the current time t is the release time of the next pulse in the pulse code sequence 2;
calculating the distance s between two pulse code sequences at the current moment tWISIThe calculation formula is as follows:
Figure BDA00021408734100000510
Figure BDA00021408734100000511
Figure BDA0002140873410000061
Figure BDA0002140873410000062
wherein,
Figure BDA0002140873410000063
for the time of the next pulse in the n coded pulse sequences at the current time t,
Figure BDA0002140873410000064
the time of the previous pulse in the n coded pulse sequences for the current time t,
Figure BDA0002140873410000065
the distance between the current time t and the time of the next pulse in the pulse code sequence,
Figure BDA0002140873410000066
the previous pulse emitting time in the nth pulse code sequence is the current time t.
On the basis of the technical scheme, the tracking of the target in the input image by using the reconstructed impulse neural network specifically comprises the following steps:
training the reconstructed impulse neural network by using a training set to obtain a trained impulse neural network;
selecting a first frame image in the input image as a template frame, and selecting a target frame area on the input image;
when the current image frame is processed, selecting 3 areas around the area where the target is located in the previous image frame as sub candidate frames;
and performing predictive identification on the template frame and the sub candidate frame by using the trained pulse neural network to obtain three score responsivity matrixes, selecting the score responsivity matrix with the maximum responsivity value, performing interpolation by a bicubic interpolation method, determining the offset of the responsivity value from the central region of the input image, obtaining the position of the target, and completing the tracking of the target in the input image.
The invention provides a short-range tracking system based on a pulse neural network, which comprises:
the encoding module is used for carrying out pulse encoding on the input image based on the attention mechanism;
the construction module is used for modifying the structure of the convolutional neural network so as to transfer the parameters of the convolutional neural network into the impulse neural network and reconstruct the impulse neural network;
the calculation module is used for calculating the pulse similarity between corresponding characteristic points in adjacent image frames of the input image to obtain the regional similarity;
a tracking module for tracking the target in the input image using the reconstructed spiking neural network.
On the basis of the technical scheme, the encoding module performs pulse encoding on the input image based on an attention mechanism, and the specific process is as follows:
carrying out convolution operation on the input image by using a 3 x 3 receptive field region operator to obtain a characteristic diagram;
based on the sequence of the characteristic values from big to small, sorting the pixel points in the characteristic graph, taking a preset number of pixel points according to the sorting, and setting the characteristic values of the taken pixel points as the characteristic values of the first ranked pixel points;
calculating the pulse distribution number s of each pixel point in the characteristic diagrami，jThe calculation formula is as follows:
Figure BDA0002140873410000071
wherein p ismaxIs the maximum pixel value, p, of a pixel point in the feature mapminIs the minimum pixel value, p, of a pixel point in the feature mapi,jThe gray value of a pixel point in the feature map is obtained, and S is the number of pulses of the feature map;
calculating the frequency f of each pixel point in the characteristic diagrami,jAnd based on the calculated pulse distribution number s of each pixel pointi，jGenerating a pulse code sequence, and calculating the frequency of each pixel point in the characteristic diagram by the following calculation formula:
fi,j＝T/si,j
wherein T is the total pulse time of the characteristic diagram.
On the basis of the technical scheme, the construction module modifies the structure of the convolutional neural network, and the specific modification process of the convolutional neural network structure comprises the following steps:
for an input layer, carrying out normalization processing on an input image;
for the convolutional layer, all offsets in the convolutional layer are set to be 0, and the sizes and initialization settings of other original cores are unchanged;
for an activation layer, the position of an activation function is originally required to be used, and the activation function is replaced by a relu () activation function;
for the Pooling layer, if the network uses the neurons with single pulse output, the original Max-Pooling layer or Average-Pooling layer is kept in the Pooling layer, and if the network uses the neurons with multi-pulse output, the Max-Pooling layer of the Pooling layer is modified into the Average-Pooling layer;
setting all offsets in the full connection layer to be 0, keeping the original neuron number and initialization of the full connection layer unchanged, and using an L2 regularization strategy for the weight of the full connection layer in a training stage; and
the erasure cannot directly represent the layer and sets the type of all weights in the convolutional neural network to a 16-bit floating point type.
On the basis of the technical scheme, the construction module migrates the convolutional neural network parameters into the impulse neural network to reconstruct the impulse neural network, and the construction of the impulse neural network structure comprises the following specific processes:
for the convolutional layer, constructing convolutional kernels with the same number and the same size as those of the convolutional layer of the convolutional neural network, and then directly transferring weight parameters of the convolutional neural network to construct the convolutional layer of the impulse neural network;
for the Pooling layer, if the network uses a single-pulse output neuron, the Max-Powing layer of the convolutional neural network corresponds to the earliest pulse emitting time in a 2 x 2 area input by the Pooling layer of the impulse neural network, and the Average-Powing layer of the convolutional neural network corresponds to the Average pulse time of the Pooling layer of the impulse neural network; if the network uses a multi-pulse output neuron, calculating the Average-Pooling layer of the Pooling layer in a convolution mode;
for the active layer, the active layer of the migration convolution neural network forms the active layer of the pulse neural network, the linear activation mode in the pulse neural network is used for calculating the accumulated voltage for the position using the relu () activation function in the migrated active layer, when the accumulated voltage reaches the release threshold, the release generates the output pulse, the membrane voltage is reset to the rest potential, when the accumulated voltage is smaller than the release threshold, the current voltage value is recorded, and when the accumulated voltage is lower than the rest potential, the membrane voltage is reset to the rest potential;
and for the full connection layer, constructing the neurons with the same number as the full connection layer of the convolutional neural network, and directly transferring the weight of the full connection layer of the convolutional neural network to form the full connection layer of the impulse neural network.
Compared with the prior art, the invention has the advantages that: the structure of the convolutional neural network is modified to transfer the parameters of the convolutional neural network into the impulse neural network, the impulse neural network is reconstructed, the reconstructed impulse neural network is combined with the strong feature extraction characteristic of the convolutional neural network and the high-efficiency calculation characteristic of the impulse neural network, the tracking accuracy is better, the resource occupation can be reduced in the tracking calculation process, and the hardware dependence is reduced
Drawings
FIG. 1 is a flowchart of a short-range tracking method based on a spiking neural network according to an embodiment of the present invention;
FIG. 2 is a block diagram of a SiamFC network;
FIG. 3 is a block diagram of a reconstructed spiking neural network.
Detailed Description
The embodiment of the invention provides a short-range tracking method based on a pulse neural network, the reconstructed pulse neural network combines the strong feature extraction characteristic of a convolutional neural network and the high-efficiency calculation characteristic of the pulse neural network, has better tracking accuracy, and can reduce the resource occupation in the tracking calculation process. The embodiment of the invention also correspondingly provides a short-range tracking system based on the impulse neural network.
Referring to fig. 1, an embodiment of the present invention provides a short-range tracking method based on a spiking neural network, including:
s1: the input image is pulse encoded based on an attention mechanism.
The encoding method in the embodiment of the invention is a pulse neural network encoding method, and is an encoding scheme based on an attention mechanism and a pulse rate. Based on an attention mechanism, carrying out pulse coding on an input image, and specifically comprising the following steps:
s101: carrying out convolution operation on the input image by using a 3 x 3 receptive field region operator to obtain a characteristic diagram; in a preferred embodiment, the receptive field region operator can be
Figure BDA0002140873410000101
Of course, in a particular application may be according toAnd adjusting the size and specific numerical value of the operator of the receptor field region volume.
S102: based on the sequence of the characteristic values from big to small, the pixel points in the characteristic diagram are sequenced, a preset number of pixel points are taken according to the sequence, the characteristic values of the taken pixel points are set as the characteristic values of the first ranked pixel points, specifically, the pixel points 20% of the first ranked pixel points can be taken, and the characteristic values of the pixel points 20% of the first ranked pixel points are set as the maximum characteristic values, so that the maximum pulse sending rate can be ensured.
S103: calculating the pulse distribution number s of each pixel point in the characteristic diagrami，jThe calculation formula is as follows:
Figure BDA0002140873410000102
wherein p ismaxIs the maximum pixel value, p, of a pixel point in the feature mapminIs the minimum pixel value, p, of a pixel point in the feature mapi,jThe gray value of a pixel point in the feature map is obtained, and S is the number of pulses of the feature map;
s104: calculating the frequency f of each pixel point in the characteristic diagrami,jAnd based on the calculated pulse distribution number s of each pixel pointi，jGenerating a pulse code sequence, and calculating the frequency of each pixel point in the characteristic diagram by the following calculation formula:
fi,j＝T/si,j
wherein T is the total pulse time of the characteristic diagram.
S2: and modifying the structure of the convolutional neural network to migrate the parameters of the convolutional neural network into the impulse neural network so as to reconstruct the impulse neural network.
In the embodiment of the invention, the structure of the convolutional neural network is modified, and the specific modification process of the structure of the convolutional neural network is as follows:
for an input layer, an input image is subjected to normalization processing, and in the normalization processing process, if a negative value is introduced due to color conversion, an abs () layer needs to be added to ensure that an input value is positive.
For the convolutional layer, all offsets in the convolutional layer are set to be 0, and the sizes and initialization settings of other original cores are unchanged;
for the active layer, the position of the active function is originally used, and the active function is replaced by a relu () active function, so that the loss of precision after the conversion is reduced by introducing negative numbers subsequently is avoided. In one case, if there is no active layer behind the convolutional layer or the fully-connected layer in the structure of the convolutional neural network, it is necessary to add an active layer using the relu () activation function behind the convolutional neural network after the structure modification.
For the Pooling layer, if the network uses the neurons with single pulse output, the original Max-Pooling layer or Average-Pooling layer is kept in the Pooling layer, and if the network uses the neurons with multi-pulse output, the Max-Pooling layer of the Pooling layer is modified into the Average-Pooling layer;
for the full-connection layer, setting all the offsets in the full-connection layer to be 0, keeping the original neuron number and initialization of the full-connection layer unchanged, and using an L2 regularization strategy for the weight of the full-connection layer in a training stage, so that the weight convergence is accelerated to a smaller range; and
deleting layers which cannot be directly represented, and setting the types of all weights in the convolutional neural network to be floating point types with 16 bits, so that the calculation efficiency after conversion is improved, and the resource occupation is reduced. Layers such as LRN layer, BN layer, etc. cannot be directly represented.
Migrating the parameters of the convolutional neural network into the impulse neural network, reconstructing the impulse neural network, and constructing an impulse neural network structure by the specific process:
for the convolutional layer, constructing convolutional kernels with the same number and the same size as those of the convolutional layer of the convolutional neural network, and then directly transferring weight parameters of the convolutional neural network to construct the convolutional layer of the impulse neural network;
for the Pooling layer, if the network uses a single pulse output neuron, the Max-Pooling layer of the convolutional neural network corresponds to the earliest pulse emitting time in the 2 x 2 area of the input of the Pooling layer of the impulse neural network, and the Average-Pooling layer of the convolutional neural networkAverage pulse time corresponding to the spiking neural network pooling layer; if the network uses a multi-pulse output neuron, calculating the Average-Pooling layer of the Pooling layer in a convolution mode; the specific process of the convolution mode calculation is as follows: when the pooling area is 2 multiplied by 2, the average pooling operation is realized by convolution operation with step length of 2, and the size and parameters of convolution kernel are set to be
Figure BDA0002140873410000121
The calculation process is equivalent to the calculation of the pulse convolution layer.
For the active layer, the active layer of the migration convolution neural network forms the active layer of the pulse neural network, the linear activation mode in the pulse neural network is used for calculating the accumulated voltage for the position using the relu () activation function in the migrated active layer, when the accumulated voltage reaches the release threshold, the release generates the output pulse, the membrane voltage is reset to the rest potential, when the accumulated voltage is smaller than the release threshold, the current voltage value is recorded, and when the accumulated voltage is lower than the rest potential, the membrane voltage is reset to the rest potential; the migration of the layers in the embodiment of the present invention is to migrate the layers of the modified convolutional neural network.
And for the full connection layer, constructing the neurons with the same number as the full connection layer of the convolutional neural network, and directly transferring the weight of the full connection layer of the convolutional neural network to form the full connection layer of the impulse neural network.
In the embodiment of the invention, a template construction technology is provided to modify the convolutional neural network, a migration-based template construction technology is provided to reconstruct the impulse neural network, and weight normalization operation is performed.
S3: calculating pulse similarity between corresponding feature points in adjacent image frames of the input image to obtain regional similarity; in the step of calculating the pulse similarity between corresponding feature points in adjacent image frames of the input image to obtain the region similarity, the calculation process of the pulse similarity between two feature points is as follows:
s301: calculating the distance between the current time t and the previous pulse emitting time in the pulse code sequence
Figure BDA0002140873410000131
The calculation formula is as follows:
Figure BDA0002140873410000132
wherein,
Figure BDA0002140873410000133
the next pulse in the nth pulse code sequence is issued for the current time t;
s302: calculating the distance delta t between the current time t and the next pulse sending time in the pulse code sequenceP(t) the calculation formula is:
Figure BDA0002140873410000134
wherein,
Figure BDA0002140873410000135
for the time of issuance of the previous pulse in pulse code sequence 1 for the current instant t,
Figure BDA0002140873410000136
the sending time of the previous pulse in the pulse code sequence 2 is the current time t;
s303: calculating a pulse emission time difference delta t after the current time t of two pulse code sequencesF(t) the calculation formula is:
Figure BDA0002140873410000137
wherein,
Figure BDA0002140873410000138
for the firing time of the following pulse in the pulse code sequence 1 at the current instant t,
Figure BDA0002140873410000139
the current time t is the release time of the next pulse in the pulse code sequence 2;
s304: calculating the distance s between two pulse code sequences at the current moment tWISIThe calculation formula is as follows:
Figure BDA00021408734100001310
Figure BDA00021408734100001311
Figure BDA0002140873410000141
Figure BDA0002140873410000142
wherein,
Figure BDA0002140873410000143
for the time of the next pulse in the n coded pulse sequences at the current time t,
Figure BDA0002140873410000144
the time of the previous pulse in the n coded pulse sequences for the current time t,
Figure BDA0002140873410000145
the distance between the current time t and the time of the next pulse in the pulse code sequence,
Figure BDA0002140873410000146
the previous pulse emitting time in the nth pulse code sequence is the current time t.
In the calculation process, the difference of the accurate ignition time of the pulse sequence is consideredThe pulse code in the present invention is defined as AAP pulse code
Figure BDA0002140873410000147
For the distance between the two antennas, the distance WISI,
Figure BDA0002140873410000148
for ISI distance, in the WISI definition proposed in the embodiment of the present invention, if the dissimilarity is 0, the condition must be satisfied:
Figure BDA0002140873410000149
namely, the requirements are met: the latest pulse-before-pulse emission time intervals are consistent and the emission time of the former pulse of the two sequences is the same, the accurate emission time of the pulse is considered, and the requirement for the evaluation mode can be met.
Using the ISI distance estimation, D is estimated for two pulse sequences {1,90}, {11,100} simulating 100msISI0.01, and using the WISI distance assessment improved by the invention, SWISI0.12; two pulse sequences {1,90}, {12,90} are examples, and D is the case of ISI distance estimationISI0.12, and using the improved WISI distance assessment of the present invention, SWISIAs the ISI distance is only concerned with the inter-pulse interval time and not with the specific occurrence time, the WISI distance proposed by the present invention is more suitable for performing similarity measurement of pulse characteristics, and the obtained estimation result applied to the tracking problem is more accurate.
Based on the proposed WISI distance evaluation mode, the similarity of the two pulses can be finally obtained. For two feature maps to be evaluated, the similarity of each feature point is obtained by one-to-one comparison according to the corresponding positions, and then the similarity of the regions is obtained by averaging the whole maps.
S4: and tracking the target in the input image by using the reconstructed impulse neural network. The reconstructed impulse neural network is equivalent to a convolution neural network with the modified structure and a WISI distance evaluation method fused with the impulse coding method, so as to obtain the impulse neural network reconstructed in the embodiment of the present invention, and the impulse neural network reconstructed in the embodiment of the present invention is shown in fig. 3, which is based on siamf (based on a full convolution twin network as a basic tracking algorithm), and the structure of siamf is shown in fig. 2.
The reconstructed impulse neural network is realized by adopting a Tensorflow deep learning framework, and the SimFC network is reproduced according to the convolution structure shown in the following table 1, and the impulse neural network structure is constructed according to the graph shown in FIG. 2.
TABLE 1
Figure BDA0002140873410000151
In the embodiment of the invention, the reconstructed impulse neural network is used for tracking the target in the input image, and the specific steps comprise:
s401: training the reconstructed impulse neural network by using a training set to obtain a trained impulse neural network;
s402: selecting a first frame image in the input image as a template frame, and selecting a target frame area on the input image; when the target frame area is selected, the excess area needs to be expanded to 127 × 127 size.
S403: when the current image frame is processed, 3 areas around the area where the target is located in the previous image frame are selected as sub candidate frames, and the size of each sub candidate frame is 255 × 255.
S404: and performing predictive identification on the template frame and the sub candidate frame by using the trained pulse neural network to obtain three score responsivity matrixes, selecting the score responsivity matrix with the maximum responsivity value, performing interpolation by a bicubic interpolation method, interpolating to 272 multiplied by 272, determining the offset of the responsivity value from the central area of the input image to obtain the position of the target, and completing the tracking of the target in the input image.
In the embodiment of the invention, the reconstructed impulse neural network is trained, the training set is an ILSVRC15 data set, and the OTB100 data set is selected by the test set. Training parameter setting: the batch size was 8 pictures; setting an exponential decay learning rate method, wherein the initial value is 0.01, and the decay coefficient is 0.86; the training algorithm selects a Momentum method, and the Momentum coefficient is selected to be 0.9; for faster convergence, the weights are constrained using L2 regularization; train a maximum of 50 epochs and add an early-stop strategy. In pulse coding, the coding simulation time is 200ms, and the maximum pulse rate is 0.6, namely, 120 pulses are generated at most. And (3) weight normalization, wherein when the impulse neural network is reconstructed, the weight normalization parameter is selected to be 99.9%, the voltage threshold value of each layer is set to be 1, a BN layer is used in the SiamFC, and the BN layer is not used in the Norm-SiamFC obtained after the convolution layer used in the middle of standardization.
According to the short-range tracking method based on the impulse neural network, the structure of the convolutional neural network is modified, so that parameters of the convolutional neural network are transferred to the impulse neural network, the impulse neural network is reconstructed, the reconstructed impulse neural network is combined with the strong feature extraction characteristic of the convolutional neural network and the high-efficiency calculation characteristic of the impulse neural network, the tracking accuracy is good, the resource occupation can be reduced in the tracking calculation process, the hardware dependence is reduced, the expansion of the application field of the impulse neural network can be further promoted, and a new technical method is provided for applying a complex deep learning model to edge equipment.
The invention provides a short-range tracking system based on a pulse neural network, which comprises:
the encoding module is used for carrying out pulse encoding on the input image based on the attention mechanism;
the construction module is used for modifying the structure of the convolutional neural network so as to transfer the parameters of the convolutional neural network into the impulse neural network and reconstruct the impulse neural network;
the calculation module is used for calculating the pulse similarity between corresponding characteristic points in adjacent image frames of the input image to obtain the regional similarity;
a tracking module for tracking the target in the input image using the reconstructed spiking neural network.
The encoding module performs pulse encoding on the input image based on an attention mechanism, and the specific process comprises the following steps:
carrying out convolution operation on the input image by using a 3 x 3 receptive field region operator to obtain a characteristic diagram;
based on the sequence of the characteristic values from big to small, sorting the pixel points in the characteristic graph, taking a preset number of pixel points according to the sorting, and setting the characteristic values of the taken pixel points as the characteristic values of the first ranked pixel points;
calculating the pulse distribution number s of each pixel point in the characteristic diagrami，jThe calculation formula is as follows:
Figure BDA0002140873410000171
wherein p ismaxIs the maximum pixel value, p, of a pixel point in the feature mapminIs the minimum pixel value, p, of a pixel point in the feature mapi,jThe gray value of a pixel point in the feature map is obtained, and S is the number of pulses of the feature map;
calculating the frequency f of each pixel point in the characteristic diagrami,jAnd based on the calculated pulse distribution number s of each pixel pointi，jGenerating a pulse code sequence, and calculating the frequency of each pixel point in the characteristic diagram by the following calculation formula:
fi,j＝T/si,j
wherein T is the total pulse time of the characteristic diagram.
The construction module modifies the structure of the convolutional neural network, and the specific modification process of the structure of the convolutional neural network is as follows:
for an input layer, carrying out normalization processing on an input image;
for the convolutional layer, all offsets in the convolutional layer are set to be 0, and the sizes and initialization settings of other original cores are unchanged;
for an activation layer, the position of an activation function is originally required to be used, and the activation function is replaced by a relu () activation function;
for the Pooling layer, if the network uses the neurons with single pulse output, the original Max-Pooling layer or Average-Pooling layer is kept in the Pooling layer, and if the network uses the neurons with multi-pulse output, the Max-Pooling layer of the Pooling layer is modified into the Average-Pooling layer;
setting all offsets in the full connection layer to be 0, keeping the original neuron number and initialization of the full connection layer unchanged, and using an L2 regularization strategy for the weight of the full connection layer in a training stage; and
the erasure cannot directly represent the layer and sets the type of all weights in the convolutional neural network to a 16-bit floating point type.
The construction module migrates the convolutional neural network parameters to the impulse neural network, reconstructs the impulse neural network, and for the construction of the impulse neural network structure, the concrete process is as follows:
for the convolutional layer, constructing convolutional kernels with the same number and the same size as those of the convolutional layer of the convolutional neural network, and then directly transferring weight parameters of the convolutional neural network to construct the convolutional layer of the impulse neural network;
for the Pooling layer, if the network uses a single-pulse output neuron, the Max-Powing layer of the convolutional neural network corresponds to the earliest pulse emitting time in a 2 x 2 area input by the Pooling layer of the impulse neural network, and the Average-Powing layer of the convolutional neural network corresponds to the Average pulse time of the Pooling layer of the impulse neural network; if the network uses a multi-pulse output neuron, calculating the Average-Pooling layer of the Pooling layer in a convolution mode;
for the active layer, the active layer of the migration convolution neural network forms the active layer of the pulse neural network, the linear activation mode in the pulse neural network is used for calculating the accumulated voltage for the position using the relu () activation function in the migrated active layer, when the accumulated voltage reaches the release threshold, the release generates the output pulse, the membrane voltage is reset to the rest potential, when the accumulated voltage is smaller than the release threshold, the current voltage value is recorded, and when the accumulated voltage is lower than the rest potential, the membrane voltage is reset to the rest potential;
and for the full connection layer, constructing the neurons with the same number as the full connection layer of the convolutional neural network, and directly transferring the weight of the full connection layer of the convolutional neural network to form the full connection layer of the impulse neural network.
The present invention is not limited to the above-described embodiments, and it will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements are also considered to be within the scope of the present invention. Those not described in detail in this specification are within the skill of the art.

Claims (8)

1. A short-range tracking method based on a pulse neural network is characterized by comprising the following steps:
pulse coding the input image based on an attention mechanism;
modifying the structure of the convolutional neural network to transfer the parameters of the convolutional neural network into the impulse neural network and reconstruct the impulse neural network;
calculating pulse similarity between corresponding feature points in adjacent image frames of the input image to obtain regional similarity;
tracking the target in the input image by using the reconstructed impulse neural network;
the attention-based method for pulse coding of the input image comprises the following specific steps:
carrying out convolution operation on the input image by using a 3 x 3 receptive field region operator to obtain a characteristic diagram;
based on the sequence of the characteristic values from big to small, sorting the pixel points in the characteristic graph, taking a preset number of pixel points according to the sorting, and setting the characteristic values of the taken pixel points as the characteristic values of the first ranked pixel points;
calculating the pulse distribution number s of each pixel point in the characteristic diagrami，jThe calculation formula is as follows:
Figure FDA0003295793090000011
wherein p ismaxIs the maximum pixel value, p, of a pixel point in the feature mapminIs the minimum pixel value, p, of a pixel point in the feature mapi,jThe gray value of a pixel point in the feature map is obtained, and S is the number of pulses of the feature map;
calculating the frequency f of each pixel point in the characteristic diagrami,jAnd based on the calculated pulse distribution number s of each pixel pointi，jGenerating a pulse code sequence, and calculating the frequency of each pixel point in the characteristic diagram by the following calculation formula:
fi,j＝T/si,j
wherein T is the total pulse time of the characteristic diagram.
2. The impulse neural network-based short-range tracking method of claim 1, wherein the structure of the convolutional neural network is modified by the following specific process:
for an input layer, carrying out normalization processing on an input image;
for the convolutional layer, all offsets in the convolutional layer are set to be 0, and the sizes and initialization settings of other original cores are unchanged;
for an activation layer, the position of an activation function is originally required to be used, and the activation function is replaced by a relu () activation function;
for the Pooling layer, if the network uses the neurons with single pulse output, the original Max-Pooling layer or Average-Pooling layer is kept in the Pooling layer, and if the network uses the neurons with multi-pulse output, the Max-Pooling layer of the Pooling layer is modified into the Average-Pooling layer;
setting all offsets in the full connection layer to be 0, keeping the original neuron number and initialization of the full connection layer unchanged, and using an L2 regularization strategy for the weight of the full connection layer in a training stage; and
the erasure cannot directly represent the layer and sets the type of all weights in the convolutional neural network to a 16-bit floating point type.
3. The impulse neural network-based short-range tracking method according to claim 2, wherein the convolutional neural network parameters are migrated into the impulse neural network to reconstruct the impulse neural network, and the specific process for constructing the impulse neural network structure is as follows:
for the convolutional layer, constructing convolutional kernels with the same number and the same size as those of the convolutional layer of the convolutional neural network, and then directly transferring weight parameters of the convolutional neural network to construct the convolutional layer of the impulse neural network;
for the Pooling layer, if the network uses a single-pulse output neuron, the Max-Powing layer of the convolutional neural network corresponds to the earliest pulse emitting time in a 2 x 2 area input by the Pooling layer of the impulse neural network, and the Average-Powing layer of the convolutional neural network corresponds to the Average pulse time of the Pooling layer of the impulse neural network; if the network uses a multi-pulse output neuron, calculating the Average-Pooling layer of the Pooling layer in a convolution mode;
for the active layer, the active layer of the migration convolution neural network forms the active layer of the pulse neural network, the linear activation mode in the pulse neural network is used for calculating the accumulated voltage for the position using the relu () activation function in the migrated active layer, when the accumulated voltage reaches the release threshold, the release generates the output pulse, the membrane voltage is reset to the rest potential, when the accumulated voltage is smaller than the release threshold, the current voltage value is recorded, and when the accumulated voltage is lower than the rest potential, the membrane voltage is reset to the rest potential;
and for the full connection layer, constructing the neurons with the same number as the full connection layer of the convolutional neural network, and directly transferring the weight of the full connection layer of the convolutional neural network to form the full connection layer of the impulse neural network.
4. The impulse neural network-based short-range tracking method as claimed in claim 1, wherein in the step of calculating the impulse similarity between corresponding feature points in adjacent image frames of the input image to obtain the region similarity, the impulse similarity between two feature points is calculated by:
calculating the current time t and the pulse coding sequenceDistance of the time of the previous burst in the train
Figure FDA0003295793090000031
The calculation formula is as follows:
Figure FDA0003295793090000032
wherein,
Figure FDA0003295793090000033
the next pulse in the nth pulse code sequence is issued for the current time t;
calculating the distance delta t between the current time t and the next pulse sending time in the pulse code sequenceP(t) the calculation formula is:
Figure FDA0003295793090000034
wherein,
Figure FDA0003295793090000035
for the time of issuance of the previous pulse in pulse code sequence 1 for the current instant t,
Figure FDA0003295793090000041
the sending time of the previous pulse in the pulse code sequence 2 is the current time t;
calculating a pulse emission time difference delta t after the current time t of two pulse code sequencesF(t) the calculation formula is:
Figure FDA0003295793090000042
wherein,
Figure FDA0003295793090000043
for the firing time of the following pulse in the pulse code sequence 1 at the current instant t,
Figure FDA0003295793090000044
the current time t is the release time of the next pulse in the pulse code sequence 2;
calculating the distance s between two pulse code sequences at the current moment tWISIThe calculation formula is as follows:
Figure FDA0003295793090000045
Figure FDA0003295793090000046
Figure FDA0003295793090000047
Figure FDA0003295793090000048
wherein,
Figure FDA0003295793090000049
for the time of the next pulse in the n coded pulse sequences at the current time t,
Figure FDA00032957930900000410
the time of the previous pulse in the n coded pulse sequences for the current time t,
Figure FDA00032957930900000411
the distance between the current time t and the time of the next pulse in the pulse code sequence,
Figure FDA00032957930900000412
the previous pulse emitting time in the nth pulse code sequence is the current time t.
5. The impulse neural network-based short-range tracking method as claimed in claim 1, wherein the step of tracking the target in the input image by using the reconstructed impulse neural network comprises the following specific steps:
training the reconstructed impulse neural network by using a training set to obtain a trained impulse neural network;
selecting a first frame image in the input image as a template frame, and selecting a target frame area on the input image;
when the current image frame is processed, selecting 3 areas around the area where the target is located in the previous image frame as sub candidate frames;
and performing predictive identification on the template frame and the sub candidate frame by using the trained pulse neural network to obtain three score responsivity matrixes, selecting the score responsivity matrix with the maximum responsivity value, performing interpolation by a bicubic interpolation method, determining the offset of the responsivity value from the central region of the input image, obtaining the position of the target, and completing the tracking of the target in the input image.
6. A spiking neural network-based short-range tracking system, comprising:
the encoding module is used for carrying out pulse encoding on the input image based on the attention mechanism;
the construction module is used for modifying the structure of the convolutional neural network so as to transfer the parameters of the convolutional neural network into the impulse neural network and reconstruct the impulse neural network;
the calculation module is used for calculating the pulse similarity between corresponding characteristic points in adjacent image frames of the input image to obtain the regional similarity;
a tracking module for tracking the target in the input image using the reconstructed spiking neural network;
the encoding module performs pulse encoding on an input image based on an attention mechanism, and the specific process comprises the following steps:
carrying out convolution operation on the input image by using a 3 x 3 receptive field region operator to obtain a characteristic diagram;
based on the sequence of the characteristic values from big to small, sorting the pixel points in the characteristic graph, taking a preset number of pixel points according to the sorting, and setting the characteristic values of the taken pixel points as the characteristic values of the first ranked pixel points;
calculating the pulse distribution number s of each pixel point in the characteristic diagrami，jThe calculation formula is as follows:
Figure FDA0003295793090000051
wherein p ismaxIs the maximum pixel value, p, of a pixel point in the feature mapminIs the minimum pixel value, p, of a pixel point in the feature mapi,jThe gray value of a pixel point in the feature map is obtained, and S is the number of pulses of the feature map;
calculating the frequency f of each pixel point in the characteristic diagrami,jAnd based on the calculated pulse distribution number s of each pixel pointi，jGenerating a pulse code sequence, and calculating the frequency of each pixel point in the characteristic diagram by the following calculation formula:
fi,j＝T/si,j
wherein T is the total pulse time of the characteristic diagram.
7. The impulse neural network-based short-range tracking system of claim 6, wherein the construction module modifies the structure of the convolutional neural network by a specific modification process:
for an input layer, carrying out normalization processing on an input image;
for the convolutional layer, all offsets in the convolutional layer are set to be 0, and the sizes and initialization settings of other original cores are unchanged;
for an activation layer, the position of an activation function is originally required to be used, and the activation function is replaced by a relu () activation function;
for the Pooling layer, if the network uses the neurons with single pulse output, the original Max-Pooling layer or Average-Pooling layer is kept in the Pooling layer, and if the network uses the neurons with multi-pulse output, the Max-Pooling layer of the Pooling layer is modified into the Average-Pooling layer;
setting all offsets in the full connection layer to be 0, keeping the original neuron number and initialization of the full connection layer unchanged, and using an L2 regularization strategy for the weight of the full connection layer in a training stage; and
the erasure cannot directly represent the layer and sets the type of all weights in the convolutional neural network to a 16-bit floating point type.
8. The spiking neural network-based short-range tracking system according to claim 7, wherein the building module migrates the convolutional neural network parameters into the spiking neural network to reconstruct the spiking neural network, and for the building of the spiking neural network structure, the specific process is as follows:
for the convolutional layer, constructing convolutional kernels with the same number and the same size as those of the convolutional layer of the convolutional neural network, and then directly transferring weight parameters of the convolutional neural network to construct the convolutional layer of the impulse neural network;
for the Pooling layer, if the network uses a single-pulse output neuron, the Max-Powing layer of the convolutional neural network corresponds to the earliest pulse emitting time in a 2 x 2 area input by the Pooling layer of the impulse neural network, and the Average-Powing layer of the convolutional neural network corresponds to the Average pulse time of the Pooling layer of the impulse neural network; if the network uses a multi-pulse output neuron, calculating the Average-Pooling layer of the Pooling layer in a convolution mode;
for the active layer, the active layer of the migration convolution neural network forms the active layer of the pulse neural network, the linear activation mode in the pulse neural network is used for calculating the accumulated voltage for the position using the relu () activation function in the migrated active layer, when the accumulated voltage reaches the release threshold, the release generates the output pulse, the membrane voltage is reset to the rest potential, when the accumulated voltage is smaller than the release threshold, the current voltage value is recorded, and when the accumulated voltage is lower than the rest potential, the membrane voltage is reset to the rest potential;
and for the full connection layer, constructing the neurons with the same number as the full connection layer of the convolutional neural network, and directly transferring the weight of the full connection layer of the convolutional neural network to form the full connection layer of the impulse neural network.
CN201910668450.0A 2019-07-23 2019-07-23 Short-range tracking method and system based on impulse neural network Active CN110555523B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201910668450.0A CN110555523B (en) 2019-07-23 2019-07-23 Short-range tracking method and system based on impulse neural network
PCT/CN2020/089907 WO2021012752A1 (en) 2019-07-23 2020-05-13 Spiking neural network-based short-range tracking method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910668450.0A CN110555523B (en) 2019-07-23 2019-07-23 Short-range tracking method and system based on impulse neural network

Publications (2)

Publication Number Publication Date
CN110555523A CN110555523A (en) 2019-12-10
CN110555523B true CN110555523B (en) 2022-03-29

Family

ID=68735812

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910668450.0A Active CN110555523B (en) 2019-07-23 2019-07-23 Short-range tracking method and system based on impulse neural network

Country Status (2)

Country Link
CN (1) CN110555523B (en)
WO (1) WO2021012752A1 (en)

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110555523B (en) * 2019-07-23 2022-03-29 中建三局智能技术有限公司 Short-range tracking method and system based on impulse neural network
CN111444936A (en) * 2020-01-14 2020-07-24 中南大学 Hyperspectral remote sensing image classification method based on pulse neural network
CN111460906B (en) * 2020-03-05 2023-05-26 重庆大学 Pulse neural network mode identification method and system based on ensemble learning
CN111858989B (en) * 2020-06-09 2023-11-10 西安工程大学 Pulse convolution neural network image classification method based on attention mechanism
CN112116010B (en) * 2020-09-21 2023-12-12 中国科学院自动化研究所 Classification method for ANN-SNN conversion based on membrane potential pretreatment
CN112381857A (en) * 2020-11-12 2021-02-19 天津大学 Brain-like target tracking method based on impulse neural network
CN112464807A (en) * 2020-11-26 2021-03-09 北京灵汐科技有限公司 Video motion recognition method and device, electronic equipment and storage medium
CN112633497B (en) * 2020-12-21 2023-08-18 中山大学 Convolutional impulse neural network training method based on re-weighted membrane voltage
CN112906884B (en) * 2021-02-05 2023-04-18 鹏城实验室 Brain-like prediction tracking method based on pulse continuous attractor network
CN112835844B (en) * 2021-03-03 2024-03-19 苏州蓝甲虫机器人科技有限公司 Communication sparsification method for impulse neural network calculation load
CN113159276B (en) * 2021-03-09 2024-04-16 北京大学 Model optimization deployment method, system, equipment and storage medium
CN113034542B (en) * 2021-03-09 2023-10-10 北京大学 Moving target detection tracking method
CN112953972A (en) * 2021-04-08 2021-06-11 周士博 Network intrusion detection method of single-pulse neural network time domain coding neurons
CN113435246B (en) * 2021-05-18 2024-04-05 西安电子科技大学 Intelligent radiation source individual identification method, system and terminal
CN113077017B (en) * 2021-05-24 2022-12-13 河南大学 Synthetic aperture image classification method based on pulse neural network
CN113673310B (en) * 2021-07-05 2024-06-11 西安电子科技大学 Ship tracking method based on enhanced twin network
CN113641292B (en) * 2021-07-09 2022-08-12 荣耀终端有限公司 Method and electronic equipment for operating on touch screen
CN113313119B (en) * 2021-07-30 2021-11-09 深圳市海清视讯科技有限公司 Image recognition method, device, equipment, medium and product
CN113807421B (en) * 2021-09-07 2024-03-19 华中科技大学 Feature map processing method of attention module based on pulse sending cortex model
CN113887645B (en) * 2021-10-13 2024-02-13 西北工业大学 Remote sensing image fusion classification method based on joint attention twin network
CN114037050B (en) * 2021-10-21 2022-08-16 大连理工大学 Robot degradation environment obstacle avoidance method based on internal plasticity of pulse neural network
CN114118168A (en) * 2021-12-08 2022-03-01 中国人民解放军96901部队26分队 Multi-station combined electromagnetic pulse event identification method, system and equipment
CN114282647B (en) * 2021-12-09 2024-02-02 上海应用技术大学 Pulse neural network-based target detection method for neuromorphic vision sensor
CN114489095B (en) * 2021-12-11 2023-12-26 西北工业大学 Brain-like pulse neural network control method applied to variant aircraft
CN114359200B (en) * 2021-12-28 2023-04-18 中国科学院西安光学精密机械研究所 Image definition evaluation method based on pulse coupling neural network and terminal equipment
CN114386578B (en) * 2022-01-12 2024-09-24 西安石油大学 Convolutional neural network method implemented on Hai Si NPU-free hardware
CN114519847B (en) * 2022-01-13 2024-10-01 东南大学 Target consistency judging method suitable for vehicle-road collaborative awareness system
CN114549973B (en) * 2022-01-25 2024-09-06 河南大学 Software-defined satellite-oriented hyperspectral image brain-like classification method
CN114549852B (en) * 2022-02-24 2023-04-18 四川大学 Impulse neural network training method based on color antagonism and attention mechanism
CN114627154B (en) * 2022-03-18 2023-08-01 中国电子科技集团公司第十研究所 Target tracking method deployed in frequency domain, electronic equipment and storage medium
CN114429491B (en) * 2022-04-07 2022-07-08 之江实验室 Pulse neural network target tracking method and system based on event camera
CN114708639B (en) * 2022-04-07 2024-05-14 重庆大学 FPGA chip for face recognition based on heterogeneous impulse neural network
CN114970829B (en) * 2022-06-08 2023-11-17 中国电信股份有限公司 Pulse signal processing method, device, equipment and storage
CN114972435A (en) * 2022-06-10 2022-08-30 东南大学 Target tracking method based on long-time and short-time integrated appearance updating mechanism
CN115586254B (en) * 2022-09-30 2024-05-03 陕西师范大学 Method and system for identifying metal material based on convolutional neural network
CN115723280B (en) * 2022-12-07 2023-05-23 河北科技大学 Polyimide film production equipment with adjustable thickness
CN117237604B (en) * 2023-09-14 2024-08-13 电子科技大学重庆微电子产业技术研究院 Target tracking method, device, computer equipment and storage medium
CN117314972B (en) * 2023-11-21 2024-02-13 安徽大学 Target tracking method of pulse neural network based on multi-class attention mechanism
CN118072079A (en) * 2024-01-29 2024-05-24 中国科学院自动化研究所 Small target object identification method and device based on impulse neural network

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346489A (en) * 2010-07-28 2012-02-08 中国科学院自动化研究所 Pulse neural network based method for controlling object tracking of robot
CN106250981A (en) * 2015-06-10 2016-12-21 三星电子株式会社 The impulsive neural networks of bandwidth consumption in minimizing memory access and network
CN106407990A (en) * 2016-09-10 2017-02-15 天津大学 Bionic target identification system based on event driving
CN106845541A (en) * 2017-01-17 2017-06-13 杭州电子科技大学 A kind of image-recognizing method based on biological vision and precision pulse driving neutral net
CN107333040A (en) * 2017-07-13 2017-11-07 中国科学院半导体研究所 Bionical visual imaging and processing unit
CN108830157A (en) * 2018-05-15 2018-11-16 华北电力大学（保定） Human bodys' response method based on attention mechanism and 3D convolutional neural networks
CN109214395A (en) * 2018-08-21 2019-01-15 电子科技大学 A kind of new image representation method based on impulsive neural networks
CN109816026A (en) * 2019-01-29 2019-05-28 清华大学 The fusion structure and method of convolutional neural networks and impulsive neural networks
CN113111758A (en) * 2021-04-06 2021-07-13 中山大学 SAR image ship target identification method based on pulse neural network

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9753959B2 (en) * 2013-10-16 2017-09-05 University Of Tennessee Research Foundation Method and apparatus for constructing a neuroscience-inspired artificial neural network with visualization of neural pathways
EP3516592A4 (en) * 2016-09-19 2020-05-20 HRL Laboratories, LLC Method for object detection in digital image and video using spiking neural networks
CN107292915A (en) * 2017-06-15 2017-10-24 国家新闻出版广电总局广播科学研究院 Method for tracking target based on convolutional neural networks
US10536775B1 (en) * 2018-06-21 2020-01-14 Trustees Of Boston University Auditory signal processor using spiking neural network and stimulus reconstruction with top-down attention control
CN110555523B (en) * 2019-07-23 2022-03-29 中建三局智能技术有限公司 Short-range tracking method and system based on impulse neural network

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346489A (en) * 2010-07-28 2012-02-08 中国科学院自动化研究所 Pulse neural network based method for controlling object tracking of robot
CN106250981A (en) * 2015-06-10 2016-12-21 三星电子株式会社 The impulsive neural networks of bandwidth consumption in minimizing memory access and network
CN106407990A (en) * 2016-09-10 2017-02-15 天津大学 Bionic target identification system based on event driving
CN106845541A (en) * 2017-01-17 2017-06-13 杭州电子科技大学 A kind of image-recognizing method based on biological vision and precision pulse driving neutral net
CN107333040A (en) * 2017-07-13 2017-11-07 中国科学院半导体研究所 Bionical visual imaging and processing unit
CN108830157A (en) * 2018-05-15 2018-11-16 华北电力大学（保定） Human bodys' response method based on attention mechanism and 3D convolutional neural networks
CN109214395A (en) * 2018-08-21 2019-01-15 电子科技大学 A kind of new image representation method based on impulsive neural networks
CN109816026A (en) * 2019-01-29 2019-05-28 清华大学 The fusion structure and method of convolutional neural networks and impulsive neural networks
CN113111758A (en) * 2021-04-06 2021-07-13 中山大学 SAR image ship target identification method based on pulse neural network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"Target detection in remote sensing image based on saliency computation of spiking neural network";Yang Liu等;《 IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium》;20181105;第2865-2868页 *
"脉冲神经网络在运动目标检测中的应用";林梅燕等;《现代电子技术》;20100831(第8期);正文第1-4页 *

Also Published As

Publication number Publication date
CN110555523A (en) 2019-12-10
WO2021012752A1 (en) 2021-01-28

Similar Documents

Publication Publication Date Title
CN110555523B (en) Short-range tracking method and system based on impulse neural network
CN111400620B (en) User trajectory position prediction method based on space-time embedded Self-orientation
CN109508634B (en) Ship type identification method and system based on transfer learning
CN113688723A (en) Infrared image pedestrian target detection method based on improved YOLOv5
CN108399406A (en) The method and system of Weakly supervised conspicuousness object detection based on deep learning
CN103279958B (en) A kind of image partition method based on Spiking neural network
CN110349185B (en) RGBT target tracking model training method and device
CN111275171B (en) Small target detection method based on parameter sharing multi-scale super-division reconstruction
CN112906828A (en) Image classification method based on time domain coding and impulse neural network
JP2020119533A (en) Attention driven image segmentation learning method and learning device using at least one adaptive loss weighting valve map which is used for hd map update required for satisfying level 4 of autonomous travel vehicle, and testing method and testing device using the same
CN114780739B (en) Time sequence knowledge graph completion method and system based on time graph convolution network
CN115390164B (en) Radar echo extrapolation forecasting method and system
CN112651374B (en) Future trajectory prediction method based on social information and automatic driving system
CN112910711A (en) Wireless service flow prediction method, device and medium based on self-attention convolutional network
CN114638408B (en) Pedestrian track prediction method based on space-time information
CN116129390B (en) Lane line accurate detection method for enhancing curve perception
CN114842180A (en) Point cloud completion method, device, equipment and medium
CN114419884B (en) Self-adaptive signal control method and system based on reinforcement learning and phase competition
CN116958557A (en) Three-dimensional indoor scene semantic segmentation method based on residual impulse neural network
CN115331460A (en) Large-scale traffic signal control method and device based on deep reinforcement learning
CN113553918B (en) Machine ticket issuing character recognition method based on pulse active learning
CN114004152A (en) Multi-wind-field wind speed space-time prediction method based on graph convolution and recurrent neural network
CN116188870A (en) Steel surface defect image classification method based on pulse convolution neural network
CN116721206A (en) Real-time indoor scene vision synchronous positioning and mapping method
CN114942480B (en) Ocean station wind speed forecasting method based on information perception attention dynamic cooperative network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant