CN115994563B - Brain-like situation learning model construction and training method for intelligent auxiliary driving - Google Patents
Brain-like situation learning model construction and training method for intelligent auxiliary driving Download PDFInfo
- Publication number
- CN115994563B CN115994563B CN202211372963.5A CN202211372963A CN115994563B CN 115994563 B CN115994563 B CN 115994563B CN 202211372963 A CN202211372963 A CN 202211372963A CN 115994563 B CN115994563 B CN 115994563B
- Authority
- CN
- China
- Prior art keywords
- synaptic
- neural network
- brain
- neurons
- neuron
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000012549 training Methods 0.000 title claims abstract description 14
- 238000010276 construction Methods 0.000 title claims abstract description 11
- 210000002569 neuron Anatomy 0.000 claims abstract description 122
- 230000000946 synaptic effect Effects 0.000 claims abstract description 90
- 238000013528 artificial neural network Methods 0.000 claims abstract description 69
- 239000012528 membrane Substances 0.000 claims abstract description 39
- 230000002787 reinforcement Effects 0.000 claims abstract description 19
- 230000006870 function Effects 0.000 claims abstract description 18
- 210000000225 synapse Anatomy 0.000 claims description 51
- 230000006399 behavior Effects 0.000 claims description 32
- 230000000284 resting effect Effects 0.000 claims description 15
- 230000002401 inhibitory effect Effects 0.000 claims description 14
- 230000000694 effects Effects 0.000 claims description 13
- 230000002964 excitative effect Effects 0.000 claims description 12
- 230000003068 static effect Effects 0.000 claims description 11
- 230000001419 dependent effect Effects 0.000 claims description 10
- 230000001537 neural effect Effects 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 6
- 238000001208 nuclear magnetic resonance pulse sequence Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 101000831205 Danio rerio Dynein axonemal assembly factor 11 Proteins 0.000 claims description 2
- 102100024282 Dynein axonemal assembly factor 11 Human genes 0.000 claims description 2
- 241001559542 Hippocampus hippocampus Species 0.000 claims description 2
- 101000831210 Homo sapiens Dynein axonemal assembly factor 11 Proteins 0.000 claims description 2
- 230000008447 perception Effects 0.000 claims description 2
- 230000000638 stimulation Effects 0.000 claims description 2
- 238000005728 strengthening Methods 0.000 claims description 2
- 230000003313 weakening effect Effects 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 abstract description 7
- 238000006243 chemical reaction Methods 0.000 abstract description 4
- 230000009471 action Effects 0.000 description 12
- 238000010304 firing Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000002474 experimental method Methods 0.000 description 7
- 230000001242 postsynaptic effect Effects 0.000 description 5
- 241001465754 Metazoa Species 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000000971 hippocampal effect Effects 0.000 description 3
- 210000004295 hippocampal neuron Anatomy 0.000 description 3
- 230000008062 neuronal firing Effects 0.000 description 3
- 210000004205 output neuron Anatomy 0.000 description 3
- 230000003518 presynaptic effect Effects 0.000 description 3
- 230000002441 reversible effect Effects 0.000 description 3
- 230000001629 suppression Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000003542 behavioural effect Effects 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000005284 excitation Effects 0.000 description 2
- 210000001320 hippocampus Anatomy 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000001953 sensory effect Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 210000002364 input neuron Anatomy 0.000 description 1
- 230000023886 lateral inhibition Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 230000027928 long-term synaptic potentiation Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 239000003607 modifier Substances 0.000 description 1
- 210000005215 presynaptic neuron Anatomy 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 229910052710 silicon Inorganic materials 0.000 description 1
- 239000010703 silicon Substances 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a brain-like scenario learning model construction and training method for intelligent auxiliary driving, and relates to the technical field of intelligent auxiliary driving. The construction method comprises the following steps: constructing neurons of a pulse neural network by adopting an LIF neuron model; setting membrane voltages of the neurons in different states by adopting a state function equation; introducing a synaptic model into the impulse neural network by adopting STDP learning rules based on a Hubby rule; and constructing a brain-like scene learning model based on the impulse neural network according to the controller unit, the synaptic beam and the synaptic model. The invention learns the rewarding association between stimulus and reaction in the situation learning task through the reinforcement learning mechanism, thereby not only saving the cost, but also obviously reducing the power consumption.
Description
Technical Field
The invention relates to the technical field of intelligent auxiliary driving, in particular to a brain-like situation learning model construction and training method for intelligent auxiliary driving.
Background
Intelligent driving refers to a technique in which a robot assists a person in driving, and in special cases, replaces the person in driving. Intelligent driving is an important component of strategically emerging industry as an important product of industrial revolution and informatization, is an important branch in the present artificial intelligence era, and is likely to become a next-generation intelligent terminal. Currently, unmanned developments have two paths, mainly with ADAS and artificial intelligence, respectively. The intelligent driving system of the L1-L3 level mainly takes ADAS (advanced driving assistance system) as a dominant, and the core technology is an automatic control system. With the continuous perfection and development of ADAS functions and technologies, intelligent driving can be realized under a perfect matched service system based on rich whole vehicle manufacturing experience.
Among them, the ADAS function needs to rely on neural networks for real-time operation and learning. Impulse neural networks are more similar to biological neurons than traditional artificial neural networks and have higher implementation efficiency due to the nature of event-based, asynchronous processing. However, since the activation of the traditional supervised learning approach is of a non-trivial event-based nature, it cannot be transplanted onto impulse neural networks.
Therefore, there is an urgent need for a network model that can adapt a learning method to a impulse neural network.
Disclosure of Invention
The invention aims to provide a brain-like situation learning model construction and training method for intelligent auxiliary driving, which is capable of saving cost and remarkably reducing power consumption by learning rewarding association between stimulus and reaction in a situation learning task through a reinforcement learning mechanism.
In order to achieve the above object, the present invention provides the following solutions:
in a first aspect, the present invention provides a method for constructing a brain-like situation learning model for intelligent driving assistance, including:
constructing neurons of a pulse neural network by adopting an LIF neuron model;
setting membrane voltages of the neurons in different states by adopting a state function equation;
introducing a synaptic model into the impulse neural network by adopting STDP learning rules based on a Hubby rule;
a pulsed neural network is constructed based on the controller unit, the synaptic beam and the synaptic model.
Further, the LIF neuron model is specifically:
wherein ,τm Time constant for neuronal membrane; u (u) reset Is the resting potential of the neuron membrane, i.e. the membrane voltage value of the neuron in a resting state; u (u) m The voltage value of the film at the moment t; i (t) is the input signal to the LIF neuron.
Further, the state function equation is specifically:
wherein ,membrane voltage at the nth moment of the jth neuron; u (u) reset Is the resting potential of the neuronal membrane;membrane voltage at the (n-1) th moment of the jth neuron; w [ i ]][j]A weight value for the interconnection between the neuron i and the neuron j; a is that i An input value for the ith neuron; />A direct input value that replaces a single peak at the pulse rate; u (u) l Is the leakage value of the membrane potential.
Further, the method for introducing a synaptic model into the impulse neural network by adopting STDP learning rules based on the Hubby law specifically comprises the following steps:
simplifying the STDP learning rule based on the Hubbard rule to obtain a simplified STDP learning rule; the simplified operation comprises a reduction multiplier and an exponential function on the basis of preserving time sequence association;
based on the simplified STDP learning rule, inhibitory static synapses and excitatory plastic synapses are introduced in the impulse neural network.
Further, the simplified STDP learning rule specifically includes:
wherein Δw represents a correction amount of the synaptic weight; w represents the current synaptic weight; wmax and Wmin represent the maximum and minimum synaptic weights respectively; Δt is the difference in pulse arrival times between pre-and post-synapses.
Further, based on the simplified STDP learning rule, an inhibitory static synapse and an excitatory plastic synapse are introduced in the impulse neural network, specifically:
where Δt is the difference in pulse arrival times between pre-and post-synapses; τ w A time constant representing a synaptic weight update procedure; τ -, τ+ represent the time windows in which the strengthening and weakening phenomena occur, respectively; w represents the current synaptic weight; wmax and Wmin represent the set maximum and minimum synaptic weights, respectively.
Further, the construction of the impulse neural network based on the controller unit, the synaptic beam and the synaptic model specifically comprises:
constructing a connection relationship between neurons of the impulse neural network based on the synaptic beam and the synaptic model;
constructing a impulse neural network based on the controller unit and a synaptic crossover core; the synaptic crossover core is a crossover network formed by the connection relations between neurons of the impulse neural network.
Further, the constructing a connection relationship between neurons of the impulse neural network based on the synaptic beam and the synaptic model specifically includes:
all neurons in the impulse neural network are connected by adopting a synaptic beam;
constructing a WTA network in a pulsed neural network based on the synaptic model; the impulse neural network comprises an input layer, a hidden layer and an output layer; the hidden layer and the output layer are WTA networks;
connecting the neurons of each layer with the neurons of the previous layer in the input layer, the hidden layer and the output layer through an excited plastic synapse;
in the hidden layer and the output layer, the winner of each layer is connected with the neighboring neurons of the winner of the layer through the inhibitory static synapses.
Further, the controller unit includes a scheduler, a behavior pattern block, a playback pattern block, a history sequence module, and an initialization synapse module;
the scheduler is respectively connected with the behavior mode block, the playback mode block and the initialization synaptic module, and is used for generating a control signal according to the received external control instruction and sending the control signal to the playback mode block and the initialization synaptic module; and receiving a first feedback signal of the behavior mode block and a second feedback signal of the playback mode block, controlling a synaptic crossover core according to the first feedback signal and the second feedback signal; the pulse signal represents a discrete pulse sequence at a certain moment;
the behavior mode block is respectively connected with the scheduler and the history sequence module, and is used for receiving pulse signals of the scheduler and providing input for the behavior stage of the neuron according to the pulse signals; for transmitting a first feedback signal to the scheduler; and a module for transmitting the activity sequence of neurons to the history sequence module;
the playback mode block is connected with the dispatcher and the history sequence module, and is used for receiving a pulse signal of the dispatcher and a third feedback signal of the history sequence module and providing input for a playback stage of the neuron according to the pulse signal and the third feedback signal; and for sending a second feedback signal to the scheduler;
the history sequence module is connected with the behavior pattern block and the playback pattern block, and is used for receiving the activity sequence of the behavior pattern block and storing the activity sequence of two neuron target moments; a third feedback signal for transmitting the stored active sequence of the two neuron target moments to the playback mode block;
the initialization synaptic module is respectively connected with the scheduler and the synaptic crossover core, and is used for receiving a control signal of the scheduler and performing initialization weight operation on the synaptic crossover core according to the control signal.
Further, the initialization synapse module generates an initialization weight by adopting a linear feedback shift register; the initialization weight is (W) Max -W Min ) A positive random number around/2.
In a second aspect, the present invention provides a method for constructing a brain-like situation learning model for intelligent driving assistance, including:
constructing a sample data set; the sample dataset includes eight distinct triples; the triplet includes: context, location, and item;
inputting the sample data set into a brain-like scenario learning model based on a pulse neural network to perform context-related task reinforcement learning training, so as to obtain a trained brain-like scenario learning model based on the pulse neural network; the context-dependent task reinforcement learning training includes both behavior and playback modes.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the pulse neural network of the invention sets membrane voltages in different states according to the LIF neuron model, adopts a WTA network model and a simplified STDP learning rule, does not use high cost functions such as indexes or multipliers and the like for neurons and synapses, can learn rewarding association between stimulus-reactions in a situation learning task through a reinforcement learning mechanism, is a pulse neural network architecture driven by a multiplicative event based on a reinforcement learning algorithm, can be used for context-related tasks, and can not only save cost, but also remarkably reduce power consumption.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a diagram of a brain-like situation learning model for intelligent driving assistance according to an embodiment of the present invention;
FIG. 2 is a state transition diagram of the LIF neuron model of the present invention;
FIG. 3 is a graph illustrating membrane voltage for different states of LIF neurons of the present invention;
FIG. 4a is a graph showing the comparison of the time difference of pulse arrival under the original rule of the synaptic modifier of three neurons with different initial weights according to the present invention;
FIG. 4b is a graph showing the correction versus pulse arrival time difference for the original rule and the simplified STDP rule;
FIG. 5 is a framework diagram of a brain-like situation learning model of the base impulse neural network of the invention;
FIG. 6 is a schematic illustration of the stimulus combination of the present invention used in an animal context-dependent task reinforcement learning experiment;
FIG. 7 is a schematic representation of replay operations in an animal context-dependent task reinforcement learning experiment in accordance with the present invention; wherein (a) a sequence of actions awarded for the replay: (b) replaying the action sequence diagram which is not rewarded;
FIG. 8 is a graph showing the timing of firing of three layers of neurons awarded in accordance with the present invention; wherein, (a) is a neuron discharge situation map corresponding to fig. 7 (a); (b) is a neuron discharge situation map corresponding to fig. 7 (a).
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a brain-like scene learning model construction method for intelligent auxiliary driving, which is capable of saving cost and remarkably reducing power consumption by learning rewarding association between stimulus and reaction in a situation learning task through a reinforcement learning mechanism.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
As shown in FIG. 1, the invention models the hippocampal network involved in reinforcement learning in the context-dependent tasks of animals by constructing a brain-like scenario learning model based on a pulsed neural network.
The impulse neural network of the present invention mimics the CA1 region in the hippocampus, comprising an input (sensory) layer, a hidden (hippocampal) layer and an output (motor) layer, comprising a total of 16 neurons (6 inputs, 8 hidden and 2 outputs), 64 plastic excitatory synapses (synapses between 6×8 input and hidden layers, synapses between 8×2 hidden and output layers), 58 inhibitory non-plastic synapses (56 hidden layers, 2 output layers) and a controller unit.
Wherein the input layer comprises 6 neurons and provides context-location information, and the adaptive unidirectional excitation weights connect all these input neurons to all hippocampal neurons (solid line); the hidden layer includes 8 hippocampal neurons with inhibitory linkages between them (dashed lines), but without self-inhibition, and a large number of plastic excitatory synapses connect all of the hippocampal neurons with output layer neurons (solid lines). The output layer includes 2 neurons, which are also inhibitory connections themselves, similar to the hippocampal layer. The sensory layer and the output layer are WTA networks.
In constructing a brain-like scenario learning model based on a pulsed neural network, the heubias law (Hebbian) is used as a basic learning mechanism, modeled by pulse timing-dependent plasticity rules (spike-timing-dependent plasticity, STDP) in combination with active playback.
The learning method of the context-dependent task is reinforcement learning, and aims to check the behavior generated by the main neurons of the hippocampus according to the environmental attribute in the training process and carry out corresponding rewards. Finally, the "cost function" in the impulse neural network is updated by going through a series of state-behaviors. Two phases are involved in the training process: a behavior phase and a replay phase.
Based on this, the embodiment of the invention provides a brain-like scenario learning model construction method for intelligent auxiliary driving, which comprises the following steps:
step 100: and constructing neurons of the impulse neural network by adopting an LIF neuron model.
The step 100 specifically includes:
the core of neurons of the impulse neural network is constructed based on leaky integral-and-fire (LIF) neurons.
The LIF neuron model specifically comprises:
wherein ,τm Time constant for neuronal membrane; u (u) reset Is the resting potential of the neuron membrane, i.e. the membrane voltage value of the neuron in a resting state; u (u) m The voltage value of the film at the moment t; i (t) is the input signal to the LIF neuron.
Step 200: and setting the membrane voltages of the neurons in different states by adopting a state function equation.
The step 200 specifically includes:
as shown in fig. 2-3, the different cores of the neuron outputs depend on the state function equation they employ, so that in the actual learning process, each execution may enter a different state and generate a different membrane voltage output according to the different inputs.
The state function equation is specifically:
wherein ,membrane voltage at the nth moment of the jth neuron; u (u) reset Is the resting potential of the neuronal membrane;membrane voltage at the (n-1) th moment of the jth neuron; w [ i ]][j]A weight value for the interconnection between the neuron i and the neuron j; a is that i An input value for the ith neuron; />A direct input value that replaces a single peak at the pulse rate; u (u) l Is the leakage value of the membrane potential.
Specifically, in a resting state, where neurons are inactive, the active neurons are seen as approximately absent, the membrane voltage is set to a resting potential.
In the state of integration,from the membrane voltage at time n-1->Synaptic input A i Weight A of (2) i w[i][j]And a possible direct input potential +.>The sum of w [ i ]][j]Weights for the interrelation between neuron i and neuron j, < >>The LIF neurons after the integration state can transition to either the waiting state or the discharging state depending on the membrane voltage value, which is a direct input value at a pulse rate instead of a single peak.
In the waiting state of the vehicle, the vehicle is in a waiting state,from the membrane voltage at time n-1->Subtracting the leakage value u of the membrane potential l ，u l When the membrane voltage does not reach the threshold value, the neuron cells automatically leak from the membrane voltage, and the potential tends to return to the resting potential.
In the discharge state, the membrane potential exceeds its threshold u th When a peak discharge is generated, the neuron no longer receives the stimulus to maintain the resting potential, and thus the membrane potential returns to the resting value, and the neuron enters a waiting state again.
Step 300: and introducing a synaptic model into the impulse neural network by adopting STDP learning rules based on a Hubby rule.
The step 300 specifically includes:
step 301: simplifying the STDP learning rule based on the Hubbard rule to obtain a simplified STDP learning rule; the simplified operation includes a reduction multiplier and an exponential function on a timing correlation basis.
In order to implement the synaptic model, i.e. digital synapse, on hardware, the original learning rules need to be adapted accordingly. Because the original rule uses more exponential functions and multipliers, silicon with larger area is needed; meanwhile, the synaptic weight correction quantity has a larger relation with the previous synaptic weight. Accordingly, the corresponding adaptation rules require the use of reduced multipliers and exponential functions on the basis of preserving timing correlations.
The simplified STDP learning rule specifically comprises the following steps:
the comparison of the pulse arrival time difference of the simplified STDP learning rule and the original STDP rule (w=0.5) is shown in fig. 4 b.
For implementing digital synapse model calculationsThe method is as follows: each time the scheduler sends a signal to initiate a learning phase (elearning=1), the synapse is eligible for adaptation. The amount of synaptic correction is calculated from the pulse arrival time difference whenever the synapse meets the learning condition (i.e., scheduler=true). Before entering the algorithm, initializing the synaptic weights is an essential step; the initial weight of the excitatory plastic synapse in the model is (W Max -W Min ) Positive values around/2. To generate such random values, a plurality of connected configurable linear feedback shift register blocks (LFSRs) are selected.
Step 302: based on the simplified STDP learning rule, inhibitory static synapses and excitatory plastic synapses are introduced in the impulse neural network.
In particular, the synaptic model is another important component of the impulse neural network. There are two synaptic models in neural networks: inhibitory static synapses and excitatory plastic synapses;
the first type of synapses (inhibitory static synapses) provide negative effects on membrane potential while providing strong lateral inhibition between neurons, the weights of which do not change with the operation of the neural network;
the second type of synapse (excitatory plastic synapse) is able to strengthen the mode voltage of the postsynaptic neuron. The intensity of these synapses may be changed in replay mode according to the STDP learning algorithm.
Wherein, the dynamic change process of the plastic synapse is based on STDP rule. This rule varies the weights according to the time difference of arrival of the pre-and post-synaptic pulses. If the presynaptic pulse arrives a few microseconds ahead of the postsynaptic pulse, the highlighting weight is increased, which results in a positive time difference corresponding to a long-term enhancement of the synapse (long-term potentiation, LTP); conversely, a negative time difference results, corresponding to long-term inhibition (LTD) of the synapse.
The specific calculation formula of the change of the synaptic weight is as follows:
as shown in fig. 4a, the change of the synaptic weight correction of three neurons with different initial values to the pulse arrival time difference enhances the amplitude three times as much as the suppression amplitude.
Step 400: and constructing a brain-like scene learning model based on the impulse neural network according to the controller unit, the synaptic beam and the synaptic model.
The step 400 specifically includes:
step 401: and constructing a connection relationship between neurons of the impulse neural network based on the synaptic beam and the synaptic model.
Specifically, as shown in fig. 5, the step 401 specifically includes:
a: all neurons in the impulse neural network are connected by adopting a synaptic beam and are arranged in a row; in the synaptic beam, excitatory plastic synapses and inhibitory static synapses are represented by grey and black circles, respectively.
B: constructing a winner-take-all (WTA) network in a pulsed neural network based on the synaptic model; the impulse neural network comprises an input layer, a hidden layer and an output layer; the hidden layer and the output layer are both WTA networks.
C: in the input layer, the hidden layer and the output layer, neurons of each layer are connected with neurons of the previous layer through excitatory plastic synapses to amplify local activities thereof.
D: in the hidden layer and the output layer, the winner of each layer is connected with the neighboring neurons of the winner of the layer through the inhibitory static synapses.
Step 402: based on the controller unit and the synaptic crossover core, constructing a brain-like scenario learning model based on a pulse neural network; the synaptic crossover core is a crossover network formed by the connection relations between neurons of the impulse neural network.
The invention realizes the management between sequences based on the controller core; the controller unit is responsible for controlling the behavior of the system, the sequence of states, data storage and preparing initial weights for synapses.
Wherein the controller unit includes a scheduler, a behavior pattern block, a playback pattern block, a history sequence block, and an initialization synapse module.
The scheduler module controls the network sequence; the behavior mode block controls the input of the behavior stage SNN; the playback mode block manages the network parameters of the learning and playback phases; the history sequence module stores a neuron activity sequence; the initialization synapse module prepares initial weights for synapses.
Specifically, the scheduler is respectively connected with the behavior mode block, the playback mode block and the initialization synapse module, and is used for generating a control signal according to a received external control instruction and sending the control signal to the playback mode block and the initialization synapse module; and receiving a first feedback signal of the behavior mode block and a second feedback signal of the playback mode block, controlling a synaptic crossover core according to the first feedback signal and the second feedback signal; the pulse signal represents a discrete pulse sequence at a certain moment;
the behavior mode block is respectively connected with the scheduler and the history sequence module and is used for receiving pulse signals of the scheduler and providing input for the behavior stage of the neuron according to the pulse signals; for transmitting a first feedback signal to the scheduler; and a module for transmitting the activity sequence of neurons to the history sequence module;
the playback mode block is connected with the scheduler and the history sequence module, and is used for receiving a pulse signal of the scheduler and a third feedback signal of the history sequence module, and providing input for a playback stage of the neuron according to the pulse signal and the third feedback signal; and for sending a second feedback signal to the scheduler;
the history sequence module is connected with the behavior pattern block and the playback pattern block and is used for receiving the activity sequence of the behavior pattern block and storing the activity sequence of two neuron target moments; a third feedback signal for transmitting the stored active sequence of the two neuron target moments to the playback mode block;
the initialization synaptic module is respectively connected with the scheduler and the synaptic crossover core, and is used for receiving a control signal of the scheduler and performing initialization weight operation on the synaptic crossover core according to the control signal. All initial suppression and excitation values for synaptic weights are generated using a Linear Feedback Shift Register (LFSR) block. A control signal is obtained from the scheduler and initial weights are sent to the synaptic crossover cores.
Based on the above, the invention also discloses a brain-like scenario learning model training method for intelligent auxiliary driving, which comprises the following steps:
step A: constructing a sample data set; the sample dataset includes eight distinct triples; the triplet includes: context, location, and item.
The step A specifically comprises the following steps: as shown in fig. 6, two contexts A, B are set in the experiment, each with two positions (positions 1 and 2), thus yielding 4 spatial positions A1, A2, B1, B2, e.g. A1 refers to position 1 in context a; in each trial, two "cans" (items X and Y) were randomly placed in two different locations; filling these "cans" with different materials, only one of which contains a prize, such as in context a, item X contains a prize and in context B, item Y contains a prize; thus, in the model of this experiment, due to the location and context of items X and Y, eight different triples appear: the bonus group includes A1X, A2X, B1Y and B2Y, and the non-bonus group includes A1Y, A2Y, B1X and B2X. (triplet A1X indicates that in context A, item X is located at position 1).
And (B) step (B): inputting the sample data set into a brain-like scenario learning model based on a pulse neural network to perform context-related task reinforcement learning training, so as to obtain a trained brain-like scenario learning model based on the pulse neural network; the context-dependent task reinforcement learning training includes both behavior and playback modes.
The step B specifically comprises the following steps: at the beginning of the experiment, all neurons were in a resting state. The initialized synaptic kernels provide random initial weights for the plastic synapses, while constructing WTA networks by assigning stronger suppression values to the corresponding synaptic connection points. Each test of the network was started with a random input stimulus value and repeated 100 times.
Each test consisted of two different modes: behavior mode and playback mode. As shown in fig. 7 and 8. FIG. 7 is a schematic representation of replay operations in an animal context-dependent task reinforcement learning experiment of the present invention, illustrating how the learning mechanism works on forward and reverse orders. FIG. 8 is a timing diagram of firing of three layers of neurons awarded in accordance with the present invention.
In fig. 7 (a), when a certain tri-state group triggers the rewarded action "dig", the first layer (corresponding to neurons 1 and 5 in the figure), the second layer (corresponding to neurons 8 in the figure) and the third layer of neurons are sequentially discharged (corresponding to neurons 15 in the figure). By LTP rules, this order of forward firing of neurons may enhance synaptic weights.
Fig. 7 (b) is a sequence diagram of replaying an action which is not rewarded, and when a certain tri-state group triggers an action "dig" which is not rewarded, the third layer (corresponding to the 15 th neuron in the figure), the second layer (corresponding to the 10 th neuron in the figure) and the first layer of neurons are sequentially discharged (corresponding to the 1 st and 6 th neurons in the figure). By LTD rules, this order of neuronal reverse firing may cut down synaptic weights.
Fig. 8 (a) is a diagram of a neuron firing situation corresponding to fig. 7 (a), in which the vertical axis corresponds to the neuron number, the horizontal axis represents the firing time, and the presynaptic neuron firing is described as preceding the postsynaptic neuron firing, i.e., Δt >0.
Fig. 8 (b) corresponds to the graph of the neuronal firing situation shown in fig. 7 (b), describing the situation where the pre-synaptic neuronal firing is later than the post-synaptic neuronal firing, i.e. Δt <0.
When the controller unit receives the control signal run, the whole experiment is started; a single test is initiated upon receipt of the start real signal. The scheduler receives the pulse signal from the outer layer of the neural network.
At the beginning of each trial, the scheduler enables the behavior pattern; the scheduler then switches to playback mode after obtaining events from the "dig" output neurons based on pulses at the network's outer layer. The scheduler also assigns a select signal to the multiplexer, providing an input voltage to the neurons in each mode.
The operation of the network in two modes is described as follows:
behavior pattern: the network starts with randomly selected triplets (each triplet having an accompanying complementary combination of context, location and item, e.g., A2Y is a complementary triplet of A1X). Under the influence of synaptic weights, a behavioral phase may include multiple actions between triplets (if "move" output neurons are active) and eventually end with mining one of them (if "dig" output neurons are active). After each move or dig, the states of all neurons are sampled in a history sequence. The aim is to store a maximum of 2 up-to-date historical action sequences (stimulus-response pairs) at the end of the behavioral phase.
Playback mode: the scheduler triggers a control signal E-learning to the plastic burst to qualify it for learning. The playback unit provides the appropriate inputs to all neurons based on the rewards obtained and the action sequences obtained by the sampling. The sequence is replayed within a specific time window. The awarded action sequence is replayed in forward timing: such playback enhances the relevant synapses, encouraging the awarded action sequences to be selected in the future, in accordance with STDP learning rules. The sequence of un-rewarded actions is replayed in reverse timing: this suppresses the non-rewarded action sequences from being selected in the future, according to the STDP learning rules.
In summary, the pulse neural network of the invention sets membrane voltages in different states according to the LIF neuron model, adopts a WTA network model and a simplified STDP learning rule, does not use high cost functions such as indexes or multipliers and the like for neurons and synapses, can learn rewarding association between stimulus-response in a situation learning task through a reinforcement learning mechanism, is a pulse neural network architecture driven by a multiplicative event based on a reinforcement learning algorithm, and can be used for context-related tasks.
The invention firstly realizes the impulse neural network on the Xilinx Kintext-7FPGA equipment, then uses the flashing LEDs at different heights to define the stimulation triplets, and then connects the network to the robot vehicle through the perception network layer-the sea horse layer-the motion output layer; the vehicle can not only distinguish between different stimulus triples, but also learn the rewards association between the item and the context, successfully verifying the performance of the network on hardware independent of the location of the item in the environment. The network architecture achieves higher efficiency and significantly reduces cost, while also facilitating research into hardware-implemented impulse neural networks using reinforcement learning algorithms.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, the present disclosure should not be construed as limiting the invention.
Claims (9)
1. The brain-like scenario learning model construction method for intelligent auxiliary driving is characterized by comprising the following steps of:
constructing neurons of a pulse neural network by adopting an LIF neuron model;
setting membrane voltages of the neurons in different states by adopting a state function equation;
introducing a synaptic model into the impulse neural network by adopting STDP learning rules based on a Hubby rule;
according to the controller unit, the synaptic beam and the synaptic model, a brain-like scene learning model based on a pulse neural network is constructed;
the controller unit comprises a scheduler, a behavior mode block, a playback mode block, a history sequence module and an initialization synapse module;
the scheduler is respectively connected with the behavior mode block, the playback mode block and the initialization synaptic module, and is used for generating a control signal according to the received external control instruction and sending the control signal to the playback mode block and the initialization synaptic module; and receiving a first feedback signal of the behavior mode block and a second feedback signal of the playback mode block, controlling a synaptic crossover core according to the first feedback signal and the second feedback signal; the pulse signal represents a discrete pulse sequence at a certain moment;
the behavior mode block is respectively connected with the scheduler and the history sequence module, and is used for receiving pulse signals of the scheduler and providing input for the behavior stage of the neuron according to the pulse signals; for transmitting a first feedback signal to the scheduler; and a module for transmitting the activity sequence of neurons to the history sequence module;
the playback mode block is connected with the dispatcher and the history sequence module, and is used for receiving a pulse signal of the dispatcher and a third feedback signal of the history sequence module and providing input for a playback stage of the neuron according to the pulse signal and the third feedback signal; and for sending a second feedback signal to the scheduler;
the history sequence module is connected with the behavior pattern block and the playback pattern block, and is used for receiving the activity sequence of the behavior pattern block and storing the activity sequence of two neuron target moments; a third feedback signal for transmitting the stored active sequence of the two neuron target moments to the playback mode block;
the initialization synaptic module is respectively connected with the scheduler and the synaptic crossover core, and is used for receiving a control signal of the scheduler and performing initialization weight operation on the synaptic crossover core according to the control signal; the initialization synaptic module adopts a linear feedback shift register to generate an initialization weight; the initialization weight is (W) Max -W Min ) A positive random number around/2; w represents the current synaptic weight; w (W) Max 、W Min Respectively representing the maximum value and the minimum value of the set synaptic weight;
firstly, the impulse neural network is realized on Xilinx Kintext-7FPGA equipment, then, flashing LEDs at different heights are used for defining a stimulation triplet, and then, the network is connected to a robot vehicle through a perception network layer, a sea horse layer and a motion output layer; the vehicle can not only distinguish between different stimulus triples, but also learn rewards associations between items and contexts, independent of the location of the items in the environment.
2. The method for constructing brain-like scenario learning model for intelligent driving assistance according to claim 1, wherein,
the LIF neuron model specifically comprises:
wherein ,τm Time constant for neuronal membrane; u (u) reset Is the resting potential of the neuron membrane, i.e. the membrane voltage value of the neuron in a resting state; u (u) m The voltage value of the film at the moment t; i (t) is the input signal to the LIF neuron.
3. The brain-like scenario learning model construction method for intelligent driving assistance according to claim 1, wherein the state function equation is specifically:
wherein ,membrane voltage at the nth moment of the jth neuron; u (u) reset Is the resting potential of the neuronal membrane; />Membrane voltage at the (n-1) th moment of the jth neuron; w [ i ]][j]A weight value for the interconnection between the neuron i and the neuron j; a is that i An input value for the ith neuron; />A direct input value that replaces a single peak at the pulse rate; u (u) l Is the leakage value of the membrane potential.
4. The method for constructing a brain-like scenario learning model for intelligent driving assistance according to claim 1, wherein the step of introducing a synaptic model into the impulse neural network by using STDP learning rules based on a heuristics rule comprises:
simplifying the STDP learning rule based on the Hubbard rule to obtain a simplified STDP learning rule; the simplified operation comprises a reduction multiplier and an exponential function on the basis of preserving time sequence association;
based on the simplified STDP learning rule, inhibitory static synapses and excitatory plastic synapses are introduced in the impulse neural network.
5. The method for constructing a brain-like scenario learning model for intelligent driving assistance according to claim 4, wherein the simplified STDP learning rule specifically comprises:
wherein Δw represents a correction amount of the synaptic weight; w represents the current synaptic weight; w (W) Max 、W Min Respectively representing the maximum value and the minimum value of the set synaptic weight; Δt is the difference in pulse arrival times between pre-and post-synapses.
6. The method for constructing a brain-like scenario learning model for intelligent driving assistance according to claim 4, wherein the method for introducing inhibitory static synapses and excitatory plastic synapses into the impulse neural network based on the simplified STDP learning rule is as follows:
where Δt is the difference in pulse arrival times between pre-and post-synapses; τ w A time constant representing a synaptic weight update procedure; τ - 、τ + Respectively representing time windows in which strengthening and weakening phenomena occur; w represents the current synaptic weight; w (W) Max 、W Min Respectively representing the maximum value and the minimum value of the set synaptic weights.
7. The method for constructing a brain-like scenario learning model for intelligent driving assistance according to claim 4, wherein the constructing a brain-like scenario learning model based on a pulse neural network according to a controller unit, a synaptic beam and the synaptic model specifically comprises:
constructing a connection relationship between neurons of the impulse neural network based on the synaptic beam and the synaptic model;
based on the controller unit and the synaptic crossover core, constructing a brain-like scenario learning model based on a pulse neural network; the synaptic crossover core is a crossover network formed by the connection relations between neurons of the impulse neural network.
8. The method for constructing a brain-like scenario learning model for intelligent driving assistance according to claim 7, wherein the constructing a connection relationship between neurons of the impulse neural network based on the synaptic beam and the synaptic model specifically comprises:
all neurons in the impulse neural network are connected by adopting a synaptic beam;
constructing a WTA network in a pulsed neural network based on the synaptic model; the impulse neural network comprises an input layer, a hidden layer and an output layer; the hidden layer and the output layer are WTA networks;
connecting the neurons of each layer with the neurons of the previous layer in the input layer, the hidden layer and the output layer through an excited plastic synapse;
in the hidden layer and the output layer, the winner of each layer is connected with the neighboring neurons of the winner of the layer through the inhibitory static synapses.
9. The method for constructing a brain-like scenario learning model for intelligent driving assistance according to claim 1, wherein a sample data set is constructed; the sample dataset includes eight distinct triples; the triplet includes: context, location, and item;
inputting the sample data set into a brain-like scenario learning model based on a pulse neural network to perform context-related task reinforcement learning training, so as to obtain a trained brain-like scenario learning model based on the pulse neural network; the context-dependent task reinforcement learning training includes both behavior and playback modes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211372963.5A CN115994563B (en) | 2022-10-31 | 2022-10-31 | Brain-like situation learning model construction and training method for intelligent auxiliary driving |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211372963.5A CN115994563B (en) | 2022-10-31 | 2022-10-31 | Brain-like situation learning model construction and training method for intelligent auxiliary driving |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115994563A CN115994563A (en) | 2023-04-21 |
| CN115994563B true CN115994563B (en) | 2023-08-18 |
Family
ID=85989478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211372963.5A Active CN115994563B (en) | 2022-10-31 | 2022-10-31 | Brain-like situation learning model construction and training method for intelligent auxiliary driving |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115994563B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117077753B (en) * | 2023-07-17 | 2024-03-19 | 华中科技大学 | Brain-like intelligent driving control model upgrading method and brain-like intelligent driving method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107092959A (en) * | 2017-04-07 | 2017-08-25 | 武汉大学 | Hardware friendly impulsive neural networks model based on STDP unsupervised-learning algorithms |
| CN108985447A (en) * | 2018-06-15 | 2018-12-11 | 华中科技大学 | A kind of hardware pulse nerve network system |
| CN110210563A (en) * | 2019-06-04 | 2019-09-06 | 北京大学 | The study of pattern pulse data space time information and recognition methods based on Spike cube SNN |
| CN110245389A (en) * | 2019-05-25 | 2019-09-17 | 天津大学 | Spiking feedforward network hippocampus function emulation system based on FPGA |
| CN111639754A (en) * | 2020-06-05 | 2020-09-08 | 四川大学 | Neural network construction, training and recognition method and system, and storage medium |
| WO2022045961A1 (en) * | 2020-08-31 | 2022-03-03 | Agency For Science, Technology And Research | Neurosynaptic processing core with spike time dependent plasticity (stdp) learning for a spiking neural network |
| CN114600127A (en) * | 2019-09-10 | 2022-06-07 | 辉达公司 | Architecture searching method based on machine learning for neural network |
| CN115063597A (en) * | 2022-07-11 | 2022-09-16 | 北京理工大学 | Image identification method based on brain-like learning |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10169701B2 (en) * | 2015-05-26 | 2019-01-01 | International Business Machines Corporation | Neuron peripheral circuits for neuromorphic synaptic memory array based on neuron models |
| US10552731B2 (en) * | 2015-12-28 | 2020-02-04 | International Business Machines Corporation | Digital STDP synapse and LIF neuron-based neuromorphic system |
-
2022
- 2022-10-31 CN CN202211372963.5A patent/CN115994563B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107092959A (en) * | 2017-04-07 | 2017-08-25 | 武汉大学 | Hardware friendly impulsive neural networks model based on STDP unsupervised-learning algorithms |
| CN108985447A (en) * | 2018-06-15 | 2018-12-11 | 华中科技大学 | A kind of hardware pulse nerve network system |
| CN110245389A (en) * | 2019-05-25 | 2019-09-17 | 天津大学 | Spiking feedforward network hippocampus function emulation system based on FPGA |
| CN110210563A (en) * | 2019-06-04 | 2019-09-06 | 北京大学 | The study of pattern pulse data space time information and recognition methods based on Spike cube SNN |
| CN114600127A (en) * | 2019-09-10 | 2022-06-07 | 辉达公司 | Architecture searching method based on machine learning for neural network |
| CN111639754A (en) * | 2020-06-05 | 2020-09-08 | 四川大学 | Neural network construction, training and recognition method and system, and storage medium |
| WO2022045961A1 (en) * | 2020-08-31 | 2022-03-03 | Agency For Science, Technology And Research | Neurosynaptic processing core with spike time dependent plasticity (stdp) learning for a spiking neural network |
| CN115063597A (en) * | 2022-07-11 | 2022-09-16 | 北京理工大学 | Image identification method based on brain-like learning |
Non-Patent Citations (1)
| Title |
|---|
| 基于脉冲神经网络误差反传算法的脑功能模拟与机制分析;洪朝飞;《中国优秀硕士论文全文数据库》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115994563A (en) | 2023-04-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lin et al. | Reinforcement learning with hidden states | |
| Zhang et al. | A hybrid particle swarm optimization–back-propagation algorithm for feedforward neural network training | |
| US5704016A (en) | Temporal learning neural network | |
| US9558442B2 (en) | Monitoring neural networks with shadow networks | |
| KR20160136364A (en) | Cold neuron spike timing back propagation | |
| US20100076916A1 (en) | Autonomous Learning Dynamic Artificial Neural Computing Device and Brain Inspired System | |
| WO2015142503A2 (en) | Implementing a neural-network processor | |
| CN115994563B (en) | Brain-like situation learning model construction and training method for intelligent auxiliary driving | |
| KR20160138002A (en) | Training, recognition, and generation in a spiking deep belief network (dbn) | |
| US9672464B2 (en) | Method and apparatus for efficient implementation of common neuron models | |
| CA2926824A1 (en) | Implementing synaptic learning using replay in spiking neural networks | |
| CN111582445A (en) | Efficient learning system and learning method based on impulse neural network | |
| WO2015047589A2 (en) | Methods and apparatus for implementation of group tags for neural models | |
| EP3108413A2 (en) | Dynamic spatial target selection | |
| CN113614745A (en) | Spiking synapse elements for spiking neural networks | |
| KR20160047581A (en) | Methods and apparatus for implementing a breakpoint determination unit in an artificial nervous system | |
| Meyer | Evolutionary approaches to neural control in mobile robots | |
| CN116080688B (en) | Brain-inspiring-like intelligent driving vision assisting method, device and storage medium | |
| Kasabov | Evolving fuzzy neural networks: theory and applications for on-line adaptive prediction, decision making and control | |
| Madey et al. | Integration of neurocomputing and system simulation for modeling continuous improvement systems in manufacturing | |
| US20230351195A1 (en) | Neurosynaptic Processing Core with Spike Time Dependent Plasticity (STDP) Learning For a Spiking Neural Network | |
| KR102535635B1 (en) | Neuromorphic computing device | |
| Suh et al. | A novel dynamic priority-based action-selection-mechanism integrating a reinforcement learning | |
| FI103304B (en) | Associative neuron | |
| JP3729868B2 (en) | Associative neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |