KR20180007657A - Method for neural network and apparatus perform same method - Google Patents

Abstract

Disclosed are an operating method of an artificial neuron and a device for performing the method. According to an embodiment, the artificial neuron calculates a change amount of activation based on an input signal received from an input synapse, determines whether an event occurs based on the change amount of the activation, and transmits an output signal, which corresponds to an event, to an output synapse in accordance with the occurrence of the event.

Description

[0001] METHOD FOR NEURAL NETWORK AND APPARATUS PERFORM SAME METHOD [0002]

The following embodiments are directed to a method of operating an artificial neuron and an apparatus for performing the method.

Research is underway to apply human recognition methods to devices in order to solve complex or unknown problems. One of these studies is a neural network modeling human biological neurons. Neural networks use algorithms that mimic human learning abilities. The neural network can perform mapping between input patterns and output patterns through learning. In addition, the neural network may have a generalization ability to generate a relatively correct output for input patterns that were not used for learning based on the learned results.

According to one aspect, a method for a neural network includes determining a current activation of an artificial neuron based on an input signal received via a previous activation of an artificial neuron and an input synapse of the artificial neuron, ; Determining an amount of change in activation based on the current activation and an activation corresponding to an event previously generated by the artificial neuron; Determining whether a new event has occurred based on the amount of change and the threshold of the activation; And transmitting an output signal corresponding to the new event to an output synapse in response to the occurrence of the new event.

The new event may occur according to the intersection of the amount of change of the activation and the threshold value. The output signal may include a sign bit indicating the direction of intersection of the change amount and the threshold value. The method may further include receiving a threshold corresponding to a previous layer connected through the input synapse, wherein the step of determining the current activation includes receiving a signal corresponding to the previous activation, the input signal, And determining the current activation based on the threshold.

The output signal may comprise a variation of the activation approximated by a predetermined bit precision. The bit precision can be adjusted according to at least one of the required accuracy and the available resources. The threshold may be adjusted according to the number of events occurring during a unit time. The threshold may be increased as the number of events occurring during the unit time exceeds a predetermined first threshold and may decrease as the number of events occurring during the unit time is less than a predetermined second threshold have. The first threshold may be greater than the second threshold.

The method may further comprise, after transmission of the output signal, updating the threshold based on the current activation. The threshold may be adjusted according to a fixed step, a logarithmic step, and an order of magnitude. The method may further include storing the current activation.

The method includes receiving a control signal indicating a predetermined operating mode; Receiving a framed input signal corresponding to the operation mode through the input synapse; And determining the current activation based on the framed input signal.

The neural network may be an artificial neural network, a fully connected network, a deep convolutional network, a recurrent neural network, and a spiking neural network. network).

According to another aspect, a method for a neural network includes determining a current activation of an artificial neuron based on a previous activation of an artificial neuron and an input signal received through an input synapse of the artificial neuron; Determining whether an event is generated based on a first cluster to which the previous activation belongs and a second cluster to which the current activation belongs; And transmitting an output signal corresponding to the event to an output synapse in response to the occurrence of the event.

The event may occur according to a difference between the first cluster and the second cluster. The output signal may include at least one change bit indicating at least one of a change direction of the cluster and a change amount of the cluster.

According to one aspect, a method for a recurrent neural network includes obtaining an input delta vector at t based on a difference between an input vector at t-1 and an input vector at t; obtaining a hidden state delta vector at t-1 based on the difference between the hidden state vector at t-2 and the hidden state vector at t-1; Based on the product of the product of the input delta vector at the obtained t and the weight for the input vector and the weight for the hidden state delta vector and the hidden state vector at the obtained t-1, Determining parameters; And determining a hidden state vector at t based on the determined parameters of the recurrent neural network. Here, t represents a time stamp and is an integer greater than one.

The parameters of the recurrent neural network may include at least one of a value of a reset gate, a value of an update gate, and a value of an output disable state vector.

Wherein the step of acquiring the input delta vector includes: if the difference between the input vector at t-1 and the input vector at t is greater than a predetermined threshold, the difference between the input vector at t-1 and the input vector at t Determining the input delta vector as the input delta vector; And determining the zero vector as the input delta vector if the difference between the input vector at t-1 and the input vector at t is less than a predetermined threshold.

Wherein obtaining the input delta vector comprises: obtaining a reference vector at t-1 based on the input delta vector at t-1; And obtaining the input delta vector based on the difference between the reference vector at t-1 and the input vector at t.

Wherein the step of acquiring the reference vector at t-1 comprises: if the input delta vector at t-1 is greater than the predetermined threshold, calculating the input delta vector at t-1 as a reference vector at t-1 Determining; And determining the reference vector at t-2 as the reference vector at t-1 if the input delta vector at t-1 is less than the predetermined threshold.

Wherein the obtaining of the hidden state delta vector comprises: if the difference between the input vector at t-1 and the input vector at t is greater than a predetermined threshold, obtaining the input delta vector at the t-1 and the t Determining a difference between the input vectors in the input vector; And determining the input delta vector as a zero vector when the difference between the input vector at t-1 and the input vector at t is less than a predetermined threshold value.

According to one aspect, an electronic device includes processing units corresponding to artificial neurons, each of the processing units including a memory and a processor storing instructions readable by the computer, wherein when the instructions are executed on the processor , The processor determines a current activation based on an input signal received over a previous activation and an input link and determines an amount of change in activation based on the current activation and an activation corresponding to a previously occurring event, And determines an occurrence of a new event based on the threshold value and transmits an output signal corresponding to the new event to an output link in response to the occurrence of the new event.

The processor may further receive a threshold corresponding to a previous layer connected via the input link and may determine the current activation based on the threshold corresponding to the previous activation, the input signal, and the previous layer. The processor may update the threshold based on the current activation after transmission of the output signal. The memory may store the current activation.

1 illustrates a neural network according to one embodiment.
Figure 2 shows an example of input and output of artificial neurons.
3 illustrates an event determination and output signal according to one embodiment.
4 illustrates an event determination and output signal according to another embodiment;
5 illustrates adjustment of a threshold according to one embodiment;
6 illustrates a variation of a cluster according to one embodiment;
Figure 7 shows the characteristics of a standard convolutional network for processing a standard video data set;
8 shows the stability of RNN activation over time;
9 is a diagram showing a calculation result obtained through a delta network;
10 is a block diagram illustrating an electronic device according to one embodiment.
11 shows an electronic device according to another embodiment.
12 is an operational flow diagram illustrating a method of operating an artificial neuron according to one embodiment.

It is to be understood that the specific structural or functional descriptions disclosed herein may be implemented by way of example only, and that the embodiments may be embodied in various other forms and are not limited to the embodiments described herein It does not.

The terms first or second may be used to describe various components, but such terms should be understood only for the purpose of distinguishing one component from another. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the words "comprising ", etc. are intended to designate the presence of stated features, integers, steps, operations, elements, parts or combinations thereof, It is to be understood that they do not preclude the presence or addition of components, components or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

1 is a diagram illustrating a neural network according to one embodiment. Referring to FIG. 1, a neural network 100 includes collections 110-150 that include artificial neurons.

The neural network 100 operates on an event basis, thereby reducing the computational costs of learning or recognizing. The neural network 100 may be used for deep running. Deep learning is a machine learning technique for solving problems such as image or speech recognition from a big data set. Feature hierarchies can be extracted in a multi-layered neural network through supervised or unsupervised learning of deep learning. The neural network 100 may correspond to an example of a multilayer neural network. The multilayer neural network may include a fully connected network, a deep convolutional network, and a recurrent neural network. For example, a fully connected network can be used for large vocabulary continuous speech recognition, 3D object recognition, face recognition, and matching and visual classification.

The size and depth of a multilayer neural network can be very large and deep compared to a conventional neural network. The size of a neural network can be expressed as a product of the number of layers per neuron and the number of layers, and the depth of a neural network can be expressed as the number of layers per network. For example, a neural network used in recent vision applications includes 41 layers, about 143 * 10 ⁶ weights, and 31 * 10 ⁶ neurons, and 19.7 * 10 ⁹ operations Are needed. Learning of such a far-layer neural network requires a huge database and long learning time.

Computing a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) can be expensive to drive such a large neural network in a sequential manner. Recent studies have used millions of artificial neurons learned on supercomputers for days or weeks. In learning through sequential methods, all artificial neurons are always updated by new input samples, which leads to increased consumption of computing resources depending on the complexity of the network. As will be described in detail below, embodiments can operate on an event basis based on input samples, thereby greatly reducing the computing resources used to compute the output of the network.

In the neural network 100, each of sets 110 through 150 includes artificial neurons, and the artificial neurons included in each of sets 110 through 150 may be connected to other artificial neurons. The other artificial neurons may be located in the same set as the artificial neurons, or may be located in another set. The sets 110-150 may be layers, and the sets 110-150 may be referred to as layers 110-150. In this case, neural network 100 may include successive layers 110-150 that include artificial neurons. Accordingly, the neural network 100 may correspond to a multi-layer neural network.

In FIG. 1, five layers 110 to 150 are illustrated for convenience of explanation, but the neural network 100 may include various numbers of layers. The layer 110 represents an input layer, the layers 120 to 140 represent an intermediate layer or a hidden layer, and the layer 150 represents an output layer. Artificial neurons included in the layers 110 to 140 except the output layer may be connected through links for transmitting output signals to artificial neurons of the next layer. The number of links may correspond to the number of artificial neurons included in the next layer. Such a link may be referred to as a synapse.

The neural network 100 may be a feedforward network. Each artificial neuron included in the neural network 100 may be connected to all artificial neurons of the next layer so that the neural network 100 forms a fully connected network, spatial connectivity.

Each artificial neuron included in the neural network 100 may compute a linear combination for the input (x_i) of the artificial neurons contained in the previous layer. The input (x_i) is multiplied by the synaptic weight (w_i, synaptic weight). The weighted inputs can be added to each other, and the sum (y) of the weighted inputs can be expressed by Equation (1).

The sum (y) of the weighted inputs can be input to an activation function (f). For example, the activation function f may be any one of a rectified linear unit (ReLU), a sigmoid, and a hyperbolic tangent. The activation function (f) can calculate the output (o) of the artificial neuron according to equation (2).

Hereinafter, the operation of the neural network 100 will be described with reference to the artificial neuron 105, but the description of the artificial neuron 105 may be applied to other artificial neurons. Below, the previous layer of artificial neuron 105 may refer to layer 120, and the next layer of artificial neuron 105 may refer to layer 140. The link between the artificial neurons 105 and the artificial neurons included in the layer 120 may be referred to as the input synapse of the artificial neuron 105 and the artificial neurons 105 and the artificial neurons included in the layer 140 May be referred to as the output synapse of the artificial neuron 105. [

According to one aspect, when the activation of the artificial neuron 105 changes by a predetermined amount, the artificial neuron 105 can determine the occurrence of a new event. Activation may be computed based on inputs received through the input synapse of the artificial neuron 105 and on the activation function implemented for the artificial neuron 105.

The neural network 100 according to one embodiment may be referred to as a dynamic computation net (DCN). As will be described in detail below, the DCN can be applied to various types of neural networks 100, such as an artificial neural network (ANN) or a spiking neural network (SNN). If the neural network 100 is a spiking neural network, the activation of the artificial neuron 105 may be the membrane potential of the artificial neuron 105.

The artificial neuron 105 may transmit output signals corresponding to a new event to output synapses as a new event occurs. A new event can occur according to the intersection of the activation amount and the threshold value. Here, the 'intersection of the amount of change in activation and the threshold value' indicates a case where the amount of change in activation is larger than the absolute value of the threshold value. The output signals may be multiplied by the weights of the output synapses and the artificial neurons of the next layer may receive the weighted output signal of the artificial neuron 105.

According to one embodiment, the output signal may include a sign bit indicating the direction of intersection of the amount of change in activation and the threshold. A neuron network that includes artificial neurons that output a single sign bit upon occurrence of an event may be referred to as a basic DCN. In this case, the threshold of the artificial neuron 105 may be transmitted together with artificial neurons of the next layer. According to one embodiment, a plurality of artificial neurons may share the same threshold. For example, artificial neurons belonging to the same layer may have the same threshold value. In this case, the artificial neurons of the next layer can reduce the load due to threshold transmission by receiving a shared threshold between the artificial neurons of the previous layer. The artificial neurons of the next layer can determine their activation based on the received sign bit and threshold. The artificial neuron 105 may update its threshold or a threshold shared within its own layer after transmitting the output signal to the next layer. As another example, the threshold may be set differently for each predetermined group other than the layer, or the threshold value may be set differently for each artificial neuron. As another example, the entire neuron network may use the same threshold.

According to another embodiment, the output signal may comprise a variation of the activation approximated by a predetermined bit precision. A neuron network that outputs the amount of change of activation approximated with a predetermined bit precision at the time of an event may be referred to as an analog-transmission DCN (DCN). The amount of change in the approximated activation can represent a continuous value. In this case, more bits are required for the transmission of the output signal, but the activation of the artificial neurons contained in the next layer can be calculated relatively accurately, as compared to representing the amount of change in activation by the sign bit. The artificial neurons of the next layer can determine their own activation using the amount of change in the received activation, so that the threshold of the artificial neuron 105 is not transmitted.

The artificial neuron 105 may update its threshold value after transmitting the output signal to the next layer. In the case of an analog-to-transmit DCN, the amount of change in activation is transmitted so that all artificial neurons can have their own thresholds, and each artificial neuron can change its threshold to a function based on available resources, such as limited bus capacity, .

Also, in the case of the basic DCN, it is difficult for the output signal limited to the sign bit to accurately transmit the amount of change in activation that is much larger than the threshold value. On the other hand, instead of transmitting a plurality of binary events, the artificial neuron 105 of the analog-transmitting DCN can transmit a single event accurately indicating the amount of change in activation. Thus, the slope-overload problem can be solved in an analog-to-transmit DCN.

In the embodiments described above, the threshold of artificial neurons may be updated as the new event occurs. The threshold can be adjusted in a variety of ways, such as a fixed step, a logarithmic step, and an order of magnitude. For example, if the activation varies greatly relative to the threshold, the threshold also needs to be adjusted to accommodate the change in activation. The reason why the activation is changed from 149 to 150 is that the meaning of the activation may not be large compared with the case of changing from 0 to 1. In this case, the threshold value is adjusted to the log interval, so that an artificial neuron having a very large activation can generate an event only when the amount of change in the activation is sufficiently large. The neuron network whose threshold is adjusted to the log interval may be referred to as a log-stepped threshold DCN. The artificial neurons contained in the log-interval threshold DCN may operate in the basic DCN scheme or in the analog-transmission DCN scheme.

According to another aspect, the artificial neuron 105 may be classified into one of the predetermined k clusters according to its activation, and may determine the occurrence of an event in accordance with the change of the cluster to which it belongs. Here, k may be a natural number. For example, if the artificial neuron 105 belongs to the first cluster according to previous activation of the artificial neuron 105 but the artificial neuron 105 belongs to the second cluster according to the current activation of the artificial neuron 105 , The artificial neuron 105 can determine the occurrence of an event.

The artificial neuron 105 may transmit the output signals to the output synapses as the event occurs. In this case, the output signal may include at least one change bit indicating at least one of the change direction of the cluster and the change amount of the cluster. For example, when the cluster to which the artificial neuron 105 belongs is changed to the upper cluster, the artificial neuron 105 can transmit +1 which is a positive bit, and when the cluster to which the artificial neuron 105 belongs is changed to the lower cluster , The artificial neuron 105 can transmit a negative bit of -1. To indicate that the cluster has changed more than one level, the change bit may be more than two bits. A neuron network that includes artificial neurons that output at least one change bit indicating at least one of a change direction of a cluster and an amount of change of a cluster at the time of an event may be referred to as a K-level DCN.

Artificial neuron 105 may include a decoder for decoding an input signal, an encoder for generating an output signal, and a transmitter for transmitting an output signal. The encoder can generate an output signal corresponding to the event in accordance with the occurrence of the event. For example, the output signal may include a sign bit, a change amount of the approximated activation, or a change bit. In the case of a basic DCN, the transmitter can send a threshold along with the sign bit. The decoder, encoder and transmitter may be implemented with at least one processor hardware module or at least one software module.

The neural network 100 may operate according to a static configuration in which the setting value of the neural network 100 is fixed or a dynamic configuration in which the setting value of the neural network 100 is changed dynamically. The setpoint may include threshold and bit precision. In the dynamic setting, the setting value may be changed periodically, or when the predetermined condition is satisfied, or may be changed upon request. For example, the threshold may be adjusted according to the number of events occurring during a unit time. More specifically, when the number of events occurring during a unit time exceeds a predetermined first threshold, the threshold may be increased to reduce the event. In addition, when the number of events occurring during a unit time falls below a predetermined second threshold, the threshold may be decreased to increase the event. Here, the first threshold value may be larger than the second threshold value.

Dynamic changes of settings can help optimize hardware resources in actual implementations. For example, the settings can be adjusted according to at least one of the request accuracy and the available resources. Specifically, when priority is given to reducing the consumption or delay of resources, compared with the accuracy required for detection, the bit precision may be decreased or the threshold value may be increased in accordance with the priority. In a mobile electronic device, the request accuracy can be set low to detect key keywords starting a sequence of commands, and in accordance with the detection of these key words, successive words can be detected with high accuracy. The request accuracy can be set low for the remaining keywords other than the core keywords.

The artificial neurons of the neural network 100 can perform stateful operation by storing their state. More specifically, artificial neurons can store activation at the time an event occurs to calculate the amount of change in activation. As will be described in detail below, the amount of change in activation can be determined based on the current activation and the activation corresponding to the previously occurring event. In this case, the activation corresponding to the previously generated event must be stored, so that the amount of change in activation can be calculated. Artificial neurons in a traditional feedforward network perform stateless operation without storing states, and the state of all artificial neurons is reset from new input samples. Since the neural network 100 is partially updated as the event occurs, a significant amount of computation can be saved as compared to updating the entire network for every input sample.

Specifically, the cost of the computation for operating the neural network depends on the bit precision required for the architecture of the neural network and the neural network. The architecture of the network can be determined by the model of artificial neurons, the number of layers, the number of neurons per layer, and the number of synapses per layer. Regarding the cost for calculating the network parameters, a sequential update of the two-layer fully connected network is illustrated by way of example.

Assuming that the first layer contains N artificial neurons, the second layer contains M artificial neurons, the precision of b bits is used, and the complexity of neuron motion is c, sequential updates of fully connected networks For each step, N * M * b * c bits of operation are the cost for calculation. In this case, the cost for the calculation may be reduced if either of the operations is not activated, such as when the input of the artificial neuron remains unchanged or remains at zero.

Assuming that α is the ratio of the input that activates the calculation, the neural network 100 performs N * M * b * c * α operations at each step. When? = 1, the operations of N * M * b * c *? are performed. When? = 0, zero operations are performed. Further, in the neural network 100, by setting b to a small value, the cost for calculation can be reduced. If b = 1, the neural network 100 may operate with binary-value connections. There is a tradeoff between the accuracy of the network and the bit precision, but an appropriate b that satisfies both the accuracy and the bit accuracy of the network can be selected. On the other hand, real-time applications are often able to receive input with little frame-to-frame variation. For example, a plurality of identical pixels may be included between consecutive frames of the input image. Thus, redundant calculations can be performed. Neural network 100 may reduce these overlap calculations.

Neural network 100 may operate in a ventilation mode that refreshes the status of all artificial neurons when artificial neurons transmit output signals. The ventilation mode can be used to escape the accumulation of activation errors caused by noise. The ventilation mode is used when the input of the neural network 100 is a mixture of periodic full information (e.g., framed input such as an image) of all channels and update events of certain channels between frames May be suitable for the treatment sufficiently. In the normal mode, the operation according to the event is terminated according to the transmission of the output signal. In the ventilation mode, the framed input can be all processed. The operation according to the ventilation mode can be periodically processed or can be processed by request.

For example, the artificial neuron may receive a control signal indicating a predetermined operating mode (e.g., ventilation mode). In this case, the artificial neuron can receive the framed input signal corresponding to the mode of operation through the input synapse. The artificial neuron can determine its state (e.g., activation, etc.) based on the framed input signal.

Figure 2 is an illustration of input and output examples of artificial neurons. Referring to FIG. 2, artificial neurons of the previous layer 210, artificial neurons 225, and artificial neurons of the next layer 230 are shown.

The artificial neuron 225 may determine the current activation v_c of the artificial neuron 225 based on the previous activation v_p of the artificial neuron 225 and the input signals i1, i2, i3. The artificial neuron 225 may determine the current activation v_c using equation (3).

In equation (3), v_c represents the current activation, f represents the activation function, v_p represents the previous activation, i represents the sum of the input signals, and φ represents the parameters of the artificial neurons. The parameters of artificial neurons can include the biases of artificial neurons and the state of artificial neurons. The sum (i) of the input signals can be determined by the sum of the input signals (i1, i2, i3). The input signals i1, i2, i3 may be determined based on the product of the output signals of the artificial neurons of the previous layer 210 and the weights of the input synapses of the artificial neuron 225. [ After the current activation v_c has been determined, the artificial neuron 225 can store the current activation v_c.

The artificial neuron 225 can determine the amount of change in activation Δv based on the activation v_e corresponding to the event (E_P) previously generated by the current activation v_c and the artificial neuron 225. The artificial neuron 225 can determine a value obtained by subtracting the activation (v_e) from the current activation (v_c) by the change amount (? V) of the activation.

The artificial neuron 225 can determine whether or not a new event E_N is generated based on the change amount? V of the activation and the threshold value VT. Specifically, the artificial neuron 225 can determine the occurrence of a new event E_N according to the intersection of the activation amount? V and the threshold value VT. The threshold value VT may include a first threshold value in the increasing direction and a second threshold value in the decreasing direction. Thus, the first threshold may be greater than the second threshold. In this case, as the amount of change? V of the activation exceeds the first threshold or the amount of change? V of activation is reduced below the first threshold, the artificial neuron 225 determines the occurrence of a new event E_N .

The artificial neuron 225 may transmit the output signal o corresponding to the new event E_N to the output synapse in accordance with the occurrence of the new event E_N. The output signal o can be multiplied by the weight w1, w2, w3 of the output synapse. Accordingly, the artificial neurons included in the next layer 230 may receive different input signals depending on the weight of the output synapse connected to the artificial neuron 225.

As described above, according to one embodiment, the output signal o at the basic DCN may include a sign bit indicating the direction of intersection of the activation amount change DELTA v and the threshold value VT. For example, if the change amount of activation (v) exceeds the first threshold, the artificial neuron 225 may transmit a single positive bit +1 to the artificial neurons included in the next layer 230. In addition, if the amount of change in activation (v) is reduced below the second threshold, the artificial neuron 225 may transmit a single negative bit of -1 to the artificial neurons included in the next layer 230. In this case, the threshold VT may be transmitted together with the artificial neurons of the next layer 230 through the output synapse, and the artificial neurons of the next layer 230 may be transmitted with the sign bit and threshold value received from the artificial neuron 225 RTI ID = 0.0 > VT). ≪ / RTI >

According to another embodiment, the artificial neuron 225 in the analog-to-transmit DCN can approximate the change amount? V of the activation with a predetermined bit precision, and the output signal o can change the approximate activation amount v_a . For example, when the amount of change (v) of activation exceeds or falls below the first threshold, the artificial neuron 225 transmits the amount of change (v_a) of activation approximated to 16 bits to the next layer 230, Lt; RTI ID = 0.0 > artificial neurons < / RTI > In this case, the artificial neurons of the next layer 230 can determine their activation using the approximate activation variation (v_a) received from the artificial neuron 225, so the threshold VT is not transmitted.

The artificial neuron 225 may update the threshold VT after transmitting the output signal o to the artificial neurons of the next layer 230. The artificial neuron 225 may update the threshold VT based on the current activation v_c that generated the new event E_N. For example, the artificial neuron 225 may update the threshold VT with a value similar to the current activation v_c. Alternatively, the artificial neuron 225 may adjust the threshold according to various manners such as fixed spacing, logarithmic spacing, or number of digits. The adjustment of the threshold value will be described later in detail.

3 is a diagram illustrating event determination and output signals according to one embodiment. Referring to FIG. 3, an activation signal of an artificial neuron according to time and an output signal according to an event are shown in a basic DNC.

As the amount of change of the activation at time t1 exceeds the threshold value VT1, the event E1 may be generated. A single bit (for example, a signal having a logical value of 'true') indicating +1 by the output signal o may be transmitted because the amount of change of the activation exceeds the first threshold value VT1 in the increasing direction . After the output signal o is transmitted, the threshold value VT1 may be maintained or updated to the threshold value VT2.

The event E2 and the event E3 can be generated as the amount of change of the activation at the time t2 and the time t3 exceeds the threshold value VT2 and the threshold value VT3. At this time, a single bit indicating +1 can be transmitted to the output signal o. After time t2, the threshold value VT2 may remain the same or may be updated to the threshold value VT3, and after the time t3, the threshold value VT3 may be maintained or updated to the threshold value VT4.

As the change amount of the activation at time t4 is reduced to less than the threshold value VT4, event E4 can be generated. A single bit (e.g., a signal having a logic value of 'false') indicating -1 in the output signal o can be transmitted since the variation amount of the activation has decreased to less than the second threshold value VT4 in the decreasing direction have. After the output signal o is transmitted, the threshold value VT4 may be maintained or updated to the threshold value VT5.

For ease of explanation, in the example of FIG. 3, the threshold is maintained un-updated, but the threshold may be updated in response to the occurrence of the event, as described above.

4 is a diagram illustrating event determination and output signals according to another embodiment. Referring to FIG. 4, the activation signal of the artificial neuron over time and the output signal according to the event are shown in the analog transmission DCN.

As the amount of change of the activation at time t1 exceeds the threshold value VT1, the event E1 may be generated. An artificial neuron can approximate the amount of change in activation + 1 with a predetermined bit precision. The artificial neuron can transmit the change amount of the approximated activation + 1 as the output signal (o). After the output signal o is transmitted, the threshold value VT1 may be maintained or updated to the threshold value VT2.

As the amount of change of the activation at time t2 exceeds the threshold value VT2, the event E2 may be generated. An artificial neuron can approximate the variation of activation +4.2 with a predetermined bit precision. An artificial neuron can transmit an approximated change in activation amount of +4.2 to the output signal (o). After the output signal o is transmitted, the threshold value VT2 may be maintained or updated to the threshold value VT3.

As the amount of change in activation at time t3 decreases below the threshold value VT3, event E3 can be generated. The artificial neuron can approximate the variation amount of activation-1 with a predetermined bit precision. The artificial neuron can transmit a variation of -1 of the approximated activation to the output signal (o). After the output signal o is transmitted, the threshold value VT3 may be maintained or updated to the threshold value VT4.

For convenience of illustration, in the example of FIG. 4, the threshold is maintained un-updated, but the threshold may be updated in response to the occurrence of the event, as described above.

5 is a diagram illustrating adjustment of a threshold according to an embodiment. Referring to FIG. 5, there is shown an output signal according to the activation and event of artificial neurons over time in the log-interval threshold DCN.

As the amount of change of the activation at time t1 exceeds the threshold value VT1, the event E1 may be generated. Since the amount of change in the activation exceeds the first threshold VT1 in the increasing direction, the artificial neuron can send a single bit indicating +1 to the output signal o. Alternatively, the artificial neuron can approximate the amount of change in activation by +1 with a predetermined bit precision. In this case, the artificial neuron can transmit the change amount of the approximated activation + 1 as the output signal o.

After the output signal o is transmitted, the threshold value VT1 may be updated to the threshold value VT2. For example, the threshold value VT2 may be set to twice the threshold value VT1 according to the log interval. In FIG. 5, the threshold value VT1 may be 1 and the threshold value VT2 may be 2.

As the amount of change of the activation at time t2 exceeds the threshold value VT2, the event E2 may be generated. Since the amount of change in the activation exceeds the first threshold VT1 in the increasing direction, the artificial neuron can send a single bit indicating +1 to the output signal o. Alternatively, the artificial neuron can approximate the variation amount of activation +2 with a predetermined bit precision, and transmit the approximated variation amount of activation + 2 to the output signal o.

After the output signal o is transmitted, the threshold value VT2 may be updated to the threshold value VT3. For example, the threshold value VT3 may be set to be twice the threshold value VT2 according to the log interval. In FIG. 5, the threshold value VT3 may be four. As the activation increases, the threshold value also increases, so that an event can occur at a larger change amount. Since the change amount relatively small relative to the absolute amount of the activation may not be meaningful, the occurrence of the meaningless event can be suppressed by adjusting the threshold value.

6 is a diagram illustrating a variation of a cluster according to one embodiment. Referring to FIG. 6, clusters 610 through 630 are shown.

As described above, the artificial neuron N can determine the occurrence of the event E in accordance with the change of the cluster to which it belongs. More specifically, the artificial neuron N is a neuron of the artificial neuron N, based on the previous activation v_p of the artificial neuron N, and the input signal i received through the input synapse of the artificial neuron N, It is possible to determine the current activation v_c and determine whether the event E is generated based on the first cluster to which the previous activation v_p belongs and the second cluster to which the current activation v_c belongs. The artificial neuron N may determine the occurrence of the event E if the first cluster and the second cluster are different. The artificial neuron N may transmit the output signal o corresponding to the event E to the output synapse in accordance with the occurrence of the event.

Clusters 610 through 630 may each have a different range. For example, cluster 620 may range from 0 to less than 1, cluster 610 may range from 1 to less than 10, and cluster 630 may have a range of 10 or more. The cluster 620 corresponds to the first level, the cluster 610 corresponds to the second level, and the cluster 630 corresponds to the third level. In this case, the artificial neuron may belong to any one of the clusters 610 to 630 according to its current activation v_c. For example, artificial neurons with an activation of less than 1 and less than 1 may belong to cluster 620. The number of clusters and the range of clusters may be determined by the extent of activation of artificial neurons belonging to the neural network. More specifically, if the distribution of the typical activation of the learned artificial neurons is separated by 0, 1, and 10, the clusters may have a range bounded by 0, 1, and 10 as described above.

The output signal o may comprise at least one change bit indicating at least one of a change direction of the cluster and a change amount of the cluster. For example, if the cluster to which the artificial neuron (N) belongs is changed to a higher cluster, the artificial neuron can transmit a change bit indicating +1. According to the example described above, the cluster 610 corresponds to a higher cluster of the cluster 620, and the cluster 630 corresponds to a higher cluster of the cluster 610. [ Also, cluster 630 corresponds to a two-level ancestor of cluster 620. To indicate that the cluster has changed more than one level, the change bit may be more than two bits. For example, if the cluster to which the artificial neuron (N) belongs is changed to a two-level subcluster, the artificial neuron may transmit a change bit indicating -2.

More specifically, when the previous activation v_p of the artificial neuron N is 1.5 and the current activation v_c of the artificial neuron N is 10.5, the artificial neuron N moves from the cluster 610 to the cluster 630, . In this case, the artificial neuron N may transmit a change bit indicating +1 to the output signal o. Further, when the previous activation v_p of the artificial neuron N is 1.5 and the current activation v_c of the artificial neuron N is 0.5, the artificial neuron N moves from the cluster 610 to the cluster 620 . In this case, the artificial neuron N may send a change bit indicating -1 to the output signal o. Further, when the previous activation v_p of the artificial neuron N is 0.5 and the current activation v_c of the artificial neuron N is 10.5, the artificial neuron N moves from the cluster 620 to the cluster 630 . In this case, the artificial neuron N may transmit a change bit indicating +2 to the output signal o.

As described above, according to one embodiment, the DCN can be applied to the ANN or the SNN. 7 to 9, an embodiment in which the DCN is applied to a recurrent neural network (RNN), which is a type of ANN, will be described below. The following description does not limit the scope of the DCN, and the DCN may be applied to an ANN other than the RNN, or SNN, as described below.

The embodiments below propose an RNN architecture that may be referred to as a Delta network. The delta network may correspond to the DCN described above. In a delta network, each neuron can output a value only when its activation change exceeds a threshold.

Due to the combination of factors such as the ability to process big data sets, e.g., computer resources such as the GPU, and large improvements in training algorithms, the RNN can be used to process transient sequences. For example, applications such as attention-based models for natural language processing, speech recognition, and structured prediction can be implemented using RNN. The RNN is equipped with a memory and uses a gating unit such as a long short term memory (LSTM). A gated recurrent unit (GRU) can greatly improve the training process of an RNN. However, since the RNN relies heavily on the matrix operation to update the neuron activation, a large amount of resources are required to operate the RNN.

According to embodiments, to reduce the resources for driving the RNN, the characteristics of the input stream and the characteristics of the neural representation of the input stream may be used. Here, the neural expression may correspond to the activation of the neurons described above.

In general, the input to the neural network may have a high level of temporal autocorrelation. If the input slowly changes with time, it can be seen that the temporal autocorrelation is high. For example, a temporal autocorrelation may be high in a video image having a small change between frames. When a neural network processes a high level input of temporal autocorrelation, the state of the neural network can produce slow-changing activations.

Figure 7 is a diagram illustrating the characteristics of a standard convolutional network for processing a set of standard video data. Referring to FIG. 7, the state of a standard convolutional network that processes a set of standard video data may produce slowly varying activity. In FIG. 7, the activation according to the flow of time (or frame) is highly redundant.

For example, activation over time does not change much. FIG. 7 shows that when the top 1000 frames in a clip for scene recognition are applied to a standard convolutional network, any top 50 features of the top level feature vector layer may be the result of being plotted over time. In Figure 7, the peaks tend to remain relatively constant over time, so that activation is consistent rather than random activation over time.

8 is a diagram showing the stability of RNN activation over time. Fig. 8 shows the activation characteristics of the RNN to which the numerical recognition data set is applied. More specifically, the upper diagram of FIG. 8 shows the MFCC (mel-frequency cepstral coefcients) characteristic for voice numbers. The lower figure of Figure 8 shows the activation of the neural network in response to the MFCC feature.

Referring to FIG. 8, slowly varying activation features may also appear in the computation of a recurrent neural network (RNN) that processes neural input. For example, if there is a long and stable representation of the input, a change in activation over time will occur slowly, and a high level of stability in activation may occur over time.

<Concept of Delta Network>

The purpose of a delta network is to transform a dense matrix-vector product, such as a weight matrix and a state vector, into a sparse matrix-vector product that is combined with a full addition. This conversion can reduce memory access and computation. To represent this transformation, the matrix-vector product can be defined as:

In Equation (4), r represents the value of the reset gate among the parameters of the recurrent neural network. In the following, the concept of a delta network is described with reference to r, but the following description can be applied to other parameters of a recurrent neural network. In order to calculate the matrix W of size n x n and the vector x of size n according to Equation 4, n ² operations are used, n ² + n read operations are performed, and n write operations can be performed . A plurality of matrix-vector products can be considered for the long input vector sequence x _t . Here, t is 1, 2, ..., n. A plurality of matrix-vector products may be calculated recursively according to equation (5).

In equation (5),? = X _t x _t1 and r _t1 is the result obtained from the previous calculation. Therefore, the calculation cost of equation (5) at t is zero. Also, x0 = 0 and r0 = 0. Here,? Can be referred to as an input delta vector. If Δ is related to a hidden state vector, Δ can be referred to as a hidden state delta vector. If? Is a sparse vector, the form as shown in equation (5) may be advantageous in terms of calculation cost. More specifically, r _t is the sum of the costs of Δ (n operations for a vector of size n), the cost of adding the stored previous results r _{t- 1} (n operations), and the cost of the sparse matrix product WΔ n may be calculated as a weighted matrix and a share (occupancy ratio) of ² n s operation for sparse vectors). Similarly, the memory cost to compute r _t may be calculated by fetching n x s weights for matrix W, 2n values for A, n values for r _t-1 , may be determined based on storing n values.

To clarify that the computational cost can be saved even when there is a small change in x, the use of thresholds is explained. The calculation cost according to the embodiment can be expressed by Equation (6).

Also, the memory cost can be expressed by Equation (7).

이다. 따라서, 연산 속도가 10배 증가할 수 있다.If the occupancy is 10%, then according to Equation 7

to be. Therefore, the operation speed can be increased by 10 times.

에 의해 계산되는 0의 수에 의해 속도 증가(speed up)가 주어질 수 있다. 실제로, 속도 증가는 데이터 스트림에 의해 결정될 수 있다. 예를 들어, 속도 증가는 x_t 및 x_{t -1} 사이에 얼마나 많은 값이 동일하게 머무르는지에 의해 결정될 수 있다. RNN의 입력, 중간 액티베이션 값, 및 출력을 나타내는 벡터 x가 타임 스텝마다 천천히 변하는 경우, 입력 값 x_t 및 x_{t -1}은 매우 리던턴트(redundant)해질 수 있고, 낮은 점유율 s 및 이에 상응하는 증가된 속도 증가가 이루어질 수 있다.

A speed up may be given by the number of zeros computed by &lt; / RTI &gt; In fact, the rate increase can be determined by the data stream. For example, the rate increase can be determined by how much the value stays the same between x _t and x _{t -1} . If the vector x representing the input, intermediate activation value, and output of the RNN changes slowly from time step to time step, the input values x _t and x _{t -1} can be very redundant and the low occupancy rate s and the corresponding increase The increased speed can be achieved.

<Delta network GRUs>

In GRU, a matrix-vector product operation that can be replaced by a delta network operation may appear several times. In the following, the case where the delta network is applied to the GRU will be described as an embodiment related to the RNN, but the delta network can be applied to other types of RNN such as the LSTM. Equation (8) represents the parameters of the GRU. Specifically, in Equation (8), r represents a reset gate value, z represents an update gate value, c represents an output disabled state vector, and h represents an updated hidden state vector. Also, the portion indicated in bold in Equation (8) represents a matrix-vector product operation. W and x in equation (8) can be arbitrarily transposed.

In Equation (8), W _xr , W _xu , and W _xc denote weights for the input vector x, and W _hr , W _hu , and W _hc represent weights for the hidden state vector h, respectively. In the following, W _xr , W _xu , W _xc can be represented by W _x , and W _hr , W _hu , and W _hc can be represented by W _h . t can represent a timestamp. The portion indicated in bold in Equation (8) can be replaced by a delta update defined by Equation (5), which can be expressed by Equation (9).

According to Equation (9), the input vector in the _t-1 x t _-1, and on the basis of the difference between the input vector x _t at t, the input of the delta vector Δ t, and _x can be obtained, in t-2 Based on the difference between the hidden state vector h _t-2 and the hidden state vector h _t-2 at the hidden state vector h _t-2 and t-1, the hidden state delta vector Δ _h at t-1 can be obtained. Further, on the basis of the input delta vector Δ _x and input vector product between the weight W _x for x, and the product between the weight W _h for the hidden state delta vector Δ _h and hidden state vector h of the t-1 at t, The parameters r, u and c can be determined. Based on the determined parameters r, u and c, the hidden state vector ht at t can be determined.

Z _xr , z _xu , z _xc , z _hr , z _hu , and z _hc in equation (9) can be regressively defined as the stored result of the previous calculation for the input or hidden state. For example, z _xr can be expressed by Equation (10).

The operation according to Equation (10) can be applied similarly to z _xu , z _xc , z _hr , z _hu , z _hc . The initial condition at x0 is z0: = 0. Also, as shown in the above equation, a number of additional terms can be merged into a single value, including a stored full-rank pre-activation state and bias. One merged value may appear as only one or two stored vector values per gate type. For example, the stored vector value may be expressed by Equation (11).

Finally, in accordance with the conditions of the initial state described above, the stored value M can be initialized with a corrected bias. For example, for a delta network GRU, M _{r, 0} = b _r , M _{u, 0} = b _u , M _{xc, 0} = b _c , and M _hr, Can be defined.

<Approximate Calculation in Delta Networks>

The above-described equations can be designed to give exactly the same answer as the initial calculation in the network. For example, if the difference between the input vector at t-1 and the input vector at t is greater than a predetermined threshold, then the difference between the input vector at t-1 and the input vector at t can be determined as the input delta vector, and t If the difference between the input vector at -1 and the input vector at t is less than a predetermined threshold, the zero vector may be determined as the input delta vector. The hidden state delta vector can also be determined in the same way.

보다 작은 경우에 벡터-곱 연산이 생략될 수 있다. 여기서 액티베이션의 변화는 전술된 델타 벡터들에 대응될 수 있다. 이 것은 정확히 같은 결과를 생산하지는 않지만, 근사적으로 정확한 결과를 생성할 수 있다.An application approach is possible through the above equations. For example, instead of skipping the vector-product operation when the change in activation is zero,

Vector-product operation may be omitted. Where the change in activation may correspond to the delta vectors described above. This does not produce exactly the same result, but it can produce an approximate and accurate result.

9 shows a calculation result obtained through a delta network. In Figure 9, non-zero values are displayed in black. Through the weighting matrix and the scarcity of the delta vector, the effect of scarcity can be seen. In the embodiment of FIG. 9, only about 20% of the weighting matrix can be fetched if the delta vector has a share of about 20%. Further considering that the weighting matrix has a 20% occupancy, only 4% of the original weighting matrix may appear to be used for the actual calculation.

만큼 증가하는 경우, 액티베이션의 중대한 변화가 축적됨에도 불구하고 변화가 발생하지 않을 수 있다. 그러므로, 이전 타임 스텝의 메모리는, 단지 최근 타입 스텝부터의 차이를 저장하는 것이 아닌, 문턱 값을 상회하는 변화를 야기하는 최근 값을 저장하도록 설정될 수 있다. 이러한 동작은 수학식 13으로 정의될 수 있다.If a non-zero threshold is used, errors can accumulate over several time steps. For example, if the input value x _t is close to

The change may not occur even though a significant change in the activation is accumulated. Hence, the memory of the previous time step may be set to store a recent value that causes a change above the threshold, rather than just storing the difference from the last type step. This operation can be defined by Equation (13).

및

는 참조 벡터를 나타낸다. 예를 들어, t-1에서의 입력 델타 벡터 Δ_{x , t-1}이 미리 정해진 임계치

보다 큰 경우, t-1에서의 입력 벡터 x_{i, t-1}이 t-1에서의 참조 벡터

로 결정될 수 있다. t-1에서의 입력 델타 벡터 Δ_{x, t-1}이 미리 정해진 임계치

보다 작은 경우, t-2에서의 참조 벡터

가 t-1에서의 참조 벡터

로 결정될 수 있다.In Equation (13)

And

Represents a reference vector. For example, if the input delta vector? _{X , t-1 at t-1} is greater than a predetermined threshold

, The input vector x _{i, t-1} at t-1 is the reference vector at t-1

. &Lt; / RTI &gt; If the input delta vector? _{x, t-1 at t-1} is less than a predetermined threshold

, The reference vector at t-2

Is the reference vector at t-1

. &Lt; / RTI &gt;

를 계산할 때, 입력

의 현재 값 및 델타 벡터

의 최근 값 사이의 차가 이용될 수 있다. 여기서, i는 시간 t에서 벡터의 성분을 나타내고,

는 0이 아닌 값을 갖는다. 또한, 델타 변화가 소정의 문턱 값

미만이면, 델타 변화는 0으로 설정되어, 충분히 큰 변화가 0이 아닌 업데이트를 생성할 때 교정될 작은 근사 오차를 생성한다. 유사하게, 수학식 13을 통해 히든 상태 델타 벡터 Δh_i,t를 구할 수 있다.In other words, the input delta vector

, The input

And the delta vector

Lt; / RTI &gt; may be used. Where i represents the component of the vector at time t,

Has a non-zero value. In addition, if the delta variation is less than a predetermined threshold

, The delta change is set to zero, producing a small approximation error to be corrected when generating a non-zero update that is sufficiently large. Similarly, the hidden state delta vector [Delta] hi _{, t} can be obtained through Expression (13).

Hereinafter, a training method and an optimization technique for calculating a delta network model will be described. Adding constraints to the training process can yield a powerful and accelerating delta network. Constraints will be described later.

<Training method>

a) Activation of the rounding network: The computation of the thresholded delta network described above can perform a function similar to the rounding of a partially computed state. The term is set to zero in small differences, but the network can be updated in large differences. For small rounding errors that can occur when performing rounding in the training process, there are a variety of ways to enhance the network. In order to increase the accuracy, the activation rounding is performed. In this case, the network can be successfully trained and thus robust to these small rounding errors. In addition, low-precision calculations can further reduce power consumption.

은 결정성(deterministic) 및 기울기-보존형 라운딩(gradient-preserving rounding)을 사용하여 고해상도 파라미터

로부터 생성될 수 있다. 저-해상도 파라미터

은 수학식 14로 나타낼 수 있다.In other words, a low-resolution parameter of the signed fixed-point format Q _{m .f} with _m integer bits and f fractional bits

Using a deterministic and gradient-preserving rounding, the high-resolution parameters &lt; RTI ID = 0.0 &gt;

Lt; / RTI &gt; Low-resolution parameter

Can be expressed by Equation (14).

은[-2 m + f-1, 2 m + f-1]의 범위로 클리핑된(clipped)

와 수학식 15와 같은 반올림 함수를 통해 구할 수 있다.In Equation (14)

Is clipped to the range [-2 m + f-1, 2 m + f-1]

And a rounding function such as Equation (15).

은 보다 낮은 정밀도의 영향을 고려하는 출력을 생성하는데 사용되며, 작은 기울기 업데이트는 고해상도 파라미터

에서 시간에 따라 누적된다. 훈련이 끝나면

는 버려지고(discard) 이 사용된다. 시뮬레이션 결과의 파라미터가 액티베이션이 된다.In Equation 15, ∇ denotes a slope operator. In the forward pass, the low-resolution parameter

Lt; / RTI &gt; is used to generate an output that takes into account the effect of lower precision,

Are cumulatively accumulated over time. After the training

Is discarded, Is used. The parameter of the simulation result becomes activation.

b) Add Gaussian noise to network activation: once the threshold is introduced, the network must be robust to non-propagation of small changes, and large changes should be considered important. Another way to provide robustness to small changes is to add Gaussian noise to all positions with thresholded delta activation. Gaussian noise can be added as shown in Equation (16).

In equation (16),? ~ N (?,?) For each? ∈ {? xr,? hr,? xu,? hu,? xc,? hc}, and the vector of samples from the Gaussian distribution with mean? and variance? for each component of each vector. Typically, the value? Is set to zero so that the expectation is not biased. For example, E [x _t +? a _{xr] = E [x t]} . Distributed? Can be set to evaluate common round-off errors due to non-updates.

<Direct calculation in delta network model>

The addition of Gaussian noise is still not the same as the cutting operation performed on the threshold delta network. To overcome this, models can be trained directly on the delta network. By training directly in the test model, the network can be robust against the types of errors that threshold delta networks typically make.

<Changes in Activation in Sedimentation Cost>

As the network is trained using the Delta network model, the cost can be related to the delta condition and added to the overall cost. The L1 standard for Δh in a batch can be computed as a mean absolute change in delta change and the L1 standard can be adjusted by the weighting factor β. The sparse cost L may be further included in the loss function. Equation (17) represents the sparse cost L.

Here, the L1 standard is used to determine the scarceness of Δh, so a small delta update is needed. According to the embodiments, since? Is not optimized during training,? X is not a target of L1. Therefore,? X may not be included in the relational expression.

<Optimization method>

The effect of weight sparsity: The amount of sparseness in the weight matrix of a deep network after training can affect the cost savings and speed up of the calculation. The sparseness of the weighting matrix in a trained low-precision network can be very large. 0 can work multiplicatively with the delta vector to produce the much needed multiply-accumulate. Thus, by considering the effect of weight sparsity on the number of updates, the speed improvement can be achieved without loss of additional accuracy.

10 is a block diagram illustrating an electronic device according to one embodiment. 10, an electronic device 1000 includes a processor 1010 and a memory 1020. [ The neural network described above may be implemented in electronic device 1000.

The processor 1010 may include at least one of the devices described above with reference to Figures 1-9, or may perform at least one of the methods described above with respect to Figures 1-9. For example, the processor 1010 may process the operations of the artificial neurons described above. More specifically, processor 1010 may be configured to perform, for each neuron included in the neural network, an operation for determining a current activation based on an input signal received over a previous activation and input synapse, a current activation and a previously occurring event An operation for determining the amount of change in activation based on the activation corresponding to the activation event, an operation for determining whether a new event is generated based on the amount of change in the activation and the threshold value, an output corresponding to the new event at the output synapse Lt; RTI ID = 0.0 &gt; signal. &Lt; / RTI &gt;

The memory 1020 may store instructions readable by the computer. When instructions stored in memory 1020 are executed in processor 1010, processor 1010 may process the operations of the artificial neurons described above. The memory 1020 may also store data relating to the neural network described above. For example, memory 1020 may store activation of artificial neurons and weight of synapses, and the like. Memory 1020 may be volatile memory or non-volatile memory.

The processor 1010 may execute the program and control the electronic device 1020. The electronic device 1020 is connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown) and can exchange data. The electronic device 1020 may be a computing device such as a mobile phone, a smart phone, a PDA, a tablet computer, a laptop computer, a personal computer, a tablet computer, a netbook, or an electronic device such as a television, a smart television, And may include various electronic devices. In addition, the above description may be applied to the electronic device 1000, and a detailed description thereof will be omitted.

11 is a view showing an electronic device according to another embodiment. Referring to FIG. 11, an electronic device 1100 includes processing units corresponding to artificial neurons.

The processing units may correspond one-to-one to artificial neurons of the neural network described above. Each of the processing units may process an operation for its corresponding artificial neuron or may store data for its corresponding artificial neuron. The processing units are linked to each other via a link. The link may correspond to the above-described synapse, and may be limited to have a specific bit width. Each of the processing units includes a memory and a processor for storing instructions readable by the computer. For example, a processor included in each of the processing units may be implemented as an arithmetic logic unit (ALU). Each of the processing units may be connected to all of the processing units of the other layer, such as in a fully connected network, or may have limited space connectivity, as in a convolutional network.

When the instructions stored in the memory are executed in the processor, the processor determines the current activation based on the input signal received over the previous activation and input link, and determines the amount of change in activation based on the current activation and the activation corresponding to the previously occurring event Determines whether to generate a new event based on the amount of change in the activation and threshold value, and transmits an output signal corresponding to the new event to the output link according to the occurrence of the new event. The memory can store the current activation of its corresponding artificial neuron. In addition, the above description may be applied to the electronic device 1100, and a detailed description thereof will be omitted.

12 is a flowchart illustrating an operation method of an artificial neuron according to an embodiment of the present invention. Referring to FIG. 12, in step 1210, an artificial neuron determines the current activation of an artificial neuron based on a previous activation of an artificial neuron and an input signal received through an input synapse of the artificial neuron. At step 1220, the artificial neuron determines the amount of change in activation based on the current activation and an activation corresponding to a previously occurring event by the artificial neuron. At step 1230, the artificial neuron determines whether a new event has occurred, based on the amount of change and the threshold of activation. In step 1240, the artificial neuron transmits an output signal corresponding to the new event to the output synapse as the new event occurs. In addition, the above description can be applied to the operation method of the artificial neuron, and a detailed description thereof will be omitted.

The embodiments described above may be implemented in hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented within a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate Such as an array, a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the drawings, various technical modifications and variations may be applied to those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Claims (34)

1. A method for a neural network,
Determining a current activation of the artificial neuron based on an input signal received via an activation of an artificial neuron and an input synapse of the artificial neuron;
Determining an amount of change in activation based on the current activation and an activation corresponding to an event previously generated by the artificial neuron;
Determining whether a new event has occurred based on the amount of change and the threshold of the activation; And
Transmitting an output signal corresponding to the new event to an output synapse in response to the occurrence of the new event
/ RTI &gt; The method according to claim 1,
Wherein the new event occurs according to an intersection of the amount of change of the activation and the threshold. The method according to claim 1,
Wherein the output signal comprises a sign bit indicating the direction of intersection of the amount of change and the threshold. The method according to claim 1,
Further comprising receiving a threshold corresponding to a previous layer connected through the input synapse,
Wherein determining the current activation comprises:
Determining the current activation based on the previous activation, the input signal, and a threshold corresponding to the previous layer
/ RTI &gt; The method according to claim 1,
Wherein the output signal comprises a variation of the activation approximated with a predetermined bit precision. 6. The method of claim 5,
Wherein the bit precision is adjusted according to at least one of a demand accuracy and an available resource. The method according to claim 1,
Wherein the threshold is adjusted according to the number of events occurring during a unit time. 8. The method of claim 7,
Wherein the threshold is increased as the number of events occurring during the unit time exceeds a predetermined first threshold and decreases as the number of events occurring during the unit time is less than a predetermined second threshold, 1 threshold is greater than the second threshold. The method according to claim 1,
Further comprising: after the transmission of the output signal, updating the threshold based on the current activation. 10. The method of claim 9,
Wherein the threshold is adjusted according to a fixed step, a logarithmic step, or an order of magnitude. The method according to claim 1,
Storing the current activation
&Lt; / RTI &gt; The method according to claim 1,
Receiving a control signal indicating a predetermined operation mode;
Receiving a framed input signal corresponding to the operation mode through the input synapse; And
Determining the current activation based on the framed input signal
&Lt; / RTI &gt; The method according to claim 1,
The neural network
An artificial neural network, a fully connected network, a deep convolutional network, a recurrent neural network, and a spiking neural network. One, the way. 1. A method for a neural network,
Determining a current activation of the artificial neuron based on a previous activation of the artificial neuron and an input signal received through an input synapse of the artificial neuron;
Determining whether an event is generated based on a first cluster to which the previous activation belongs and a second cluster to which the current activation belongs; And
Transmitting an output signal corresponding to the event to an output synapse in response to the occurrence of the event
/ RTI &gt; 15. The method of claim 14,
Wherein the event occurs according to a difference between the first cluster and the second cluster. 15. The method of claim 14,
Wherein the output signal comprises at least one change bit indicating at least one of a change direction of the cluster and a change amount of the cluster. 1. A method for a recurrent neural network,
obtaining an input delta vector at t based on a difference between an input vector at t-1 and an input vector at t;
obtaining a hidden state delta vector at t-1 based on the difference between the hidden state vector at t-2 and the hidden state vector at t-1;
Based on the product of the product of the input delta vector at the obtained t and the weight for the input vector and the weight for the hidden state delta vector and the hidden state vector at the obtained t-1, Determining parameters; And
Determining a hidden state vector at t based on the determined parameters of the recurrent neural network
Lt; / RTI &gt;
Where t is a time stamp and is an integer greater than 1,
Way. 18. The method of claim 17,
Wherein the parameters of the recurrent neural network include at least one of a value of a reset gate, a value of an update gate, and a value of an output hidden state vector. 18. The method of claim 17,
The step of obtaining the input delta vector
If the difference between the input vector at t-1 and the input vector at t is greater than a predetermined threshold, determining the difference between the input vector at t-1 and the input vector at t to be the input delta vector ; And
Determining a zero vector as the input delta vector if the difference between the input vector at t-1 and the input vector at t is less than the predetermined threshold,
/ RTI &gt; 18. The method of claim 17,
The step of obtaining the input delta vector
obtaining a reference vector at t-1 based on the input delta vector at t-1; And
Obtaining the input delta vector based on a difference between the reference vector at t-1 and the input vector at t;
/ RTI &gt; 21. The method of claim 20,
The step of obtaining the reference vector at t-1 comprises:
If the input delta vector at t-1 is greater than a predetermined threshold, determining an input delta vector at t-1 as a reference vector at t-1; And
If the input delta vector at t-1 is smaller than the predetermined threshold, determining a reference vector at t-2 as a reference vector at t-1
/ RTI &gt; 18. The method of claim 17,
The step of obtaining the hidden state delta vector
If the difference between the input vector at t-1 and the input vector at t is greater than a predetermined threshold, determining the input delta vector as a difference between the input vector at t-1 and the input vector at t; And
If the difference between the input vector at t-1 and the input vector at t is less than the predetermined threshold, determining the input delta vector as a zero vector
/ RTI &gt; 22. A computer program stored on a medium for executing the method of any one of claims 1 to 22 in combination with hardware. Comprising processing units corresponding to artificial neurons,
Each of the processing units comprising a memory and a processor for storing instructions readable by the computer,
If the instructions are executed in the processor, the processor determines a current activation based on an input signal received over a previous activation and an input link, and determines a change in activation based on the current activation and an activation corresponding to a previously occurring event Determines whether to generate a new event based on the amount of change and threshold of the activation and sends an output signal corresponding to the new event to an output link as the new event occurs. 25. The method of claim 24,
Wherein the new event occurs according to an intersection of the amount of change of the activation and the threshold value. 25. The method of claim 24,
Wherein the output signal includes a sign bit indicating a direction of intersection of the amount of change and the threshold value. 25. The method of claim 24,
Wherein the processor further receives a threshold corresponding to a previous layer connected via the input link and determines the current activation based on the threshold corresponding to the previous activation, the input signal, and the previous layer. 25. The method of claim 24,
Wherein the output signal comprises a variation of the activation approximated by a predetermined bit precision. 29. The method of claim 28,
Wherein the bit precision is adjusted according to at least one of a request accuracy and an available resource. 25. The method of claim 24,
Wherein the threshold is adjusted according to the number of events occurring during a unit time. 31. The method of claim 30,
Wherein the threshold is increased as the number of events occurring during the unit time exceeds a predetermined first threshold and decreases as the number of events occurring during the unit time is less than a predetermined second threshold, 1 threshold is greater than the second threshold. 25. The method of claim 24,
Wherein the processor updates the threshold based on the current activation after transmission of the output signal. 33. The method of claim 32,
Wherein the threshold is adjusted according to a fixed interval, a log interval, or a number of digits. 25. The method of claim 24,
Wherein the memory stores the current activation.

Images