TW201232429A - High-speed hardware back-propagation and recurrent type artificial neural network with flexible architecture - Google Patents

Abstract

The present invention adopts a field programmable gate array (FPGA) to develop a hardware artificial neural network, so as to fully utilize the features of parallel processing and high speed in a hardware circuit, and complete the operation of learning and recall in an artificial neural network. The present invention adopts a serial circular hardware architecture as a base to perform the computation of a 3-layer and 4-layer back-propagation artificial neural and recurrent type artificial neural network and realize the calculation of a segment linear activating function by the hardware schemes. The activating function includes a sigmoid function and a hyperbolic tangent function for allowing a user to select by himself/herself. A pipelined design is adopted to enhance the speed at an output layer error calculation in a reverse calculation. A serial circular hardware architecture is adopted at a hidden layer error calculation. Owing to the development being completed on the hardware, the portability and calculation speed are easier and faster than those developed on the software, so as to be suitable for low-end embedded systems.

Description

201232429 - VI. Description of the invention: [Technical field to which the invention pertains] The present invention is characterized in that __roads are linked to each other (4) through the characteristics of a meta-surface mimic neural network, which itself is a parallel operation of a hard hardware circuit The parallel processing and high-speed features complete the neural network learning and recall operations. [Prior Art] Prior art - D. H_erstrom proposed χι_Architecture, also known as _ | (Connected Network of Adapted Processors System). Multiple data sinks are transmitted to gleInStructionst_^, all nodes are executed in the same-clock. The same instructions, as shown in Figure _. The advantage is that the accumulator and the multiplier designed by each bus of the bus (4) can make the overall operation speed of the line more efficient [however, the operation of the spine only completes the neural network part] is not kind. The activation function has no block and does not have a complete learning function. And because CNAPS is not accurate enough in many applications, and the general neural network ^ > input, weight, transfer function, etc. usually use the floating point format, it will cause a lot of errors in the operation format conversion. The prior art 2' architecture is a two-dimensional pulsation array architecture (2D_SystQlic 2D SA) as shown in FIG. The neurons (neu_) of each link of this architecture have the product of the multiplication and operation data and the weight value. The material f is obtained from the top to the bottom through each node and the node _« value, and the neuron is obtained. Output. A vertical point in this architecture can constitute a function of a neuron, but since each node contains a multiplier, it will consume a large amount of wafer area and add hardware cost. 3 201232429 . Jiaotong University's smashing place Guo _ _ VLSI Ping (four) issued __ road. The system is connected with a single operation controller, and the control side uses a form of parallel ship flow to be fine (10). It is expected that the dirty bus will require a large number of wafers, which will greatly increase the cost and the number of computing units will affect the control architecture at the same time, which will not only cause inconvenience in transplantation, but also in the instant editing. More _. Her VLSI technology is also used as a hardware development platform to design a neural network crystal # 'silk structure _ time-sharing computing architecture, double the amount of data, reducing the cost of the hardware architecture. A wafer can be processed by a number of prior art three-tie 岐. From the level (10) degree to see the operation of the neural network based on, and remember the "hardware activation function of the money check method", this development method only needs to be targeted; (4) the number of units is modified. The prior art 4 constructs a circular (four) hardware architecture for customs clearance and improves the part of the learning algorithm that many previous architectures cannot handle with hardware. Silk technology five _ parallel processing design, but the design only a large number of hardware to build the transport element, although the phase is fast in the system operation time, but it consumes a lot of hardware. The hardware architecture of the prior art six construction and segmentation calculations must be achieved only through the software side. Once the application problems are different, the hardware must be re-synthesized when the network architecture changes. It can be seen that there are still many shortcomings in the above-mentioned items, which is not a good designer, but needs to be improved. In view of the above (4) technology, the inventors of the case have improved and innovated in the past, and after years of hard work and research, they finally succeeded in researching and developing the elastic hardware back-transfer and feedback-type neural network. . 201232429 [Disclosed] The utility of the entire hardware architecture to realize various neural networks through the controller and the use of memory, and increasing the practicality of the hardware-like neural network in its application is the focus of the present invention. 1.1 Inverted-transfer-like neural network The inverse-transfer-like neural network is currently the most representative artificial neural network architecture. Its architecture is shown in Figure 3. The reverse-transfer-like neural network is the architecture of the most widely used network belonging to the multi-counter. Compared to the classical perceptron network, the mechanism of the hidden layer is added, and the smoothing and differentiating activation function is used instead. The basic principle of the multi-layer perceptron learning algorithm is based on the concept of the Gradient Steepest Descent Meth〇d. In the process of learning, the weight value of the network is adjusted to minimize the error function. The multi-layer perceptron includes an input layer, a hidden layer and an output layer. The functions of each layer are as follows: 1. Input layer: used to represent the input variables received by the neural network based on various problems, each neuron in the input layer. Representing an input variable, due to the value range of the activation function, different inputs will affect the learning effect, so the appropriate conversion is required when inputting 'The value range of the input variable is between 〇~1. 2. Hidden layer: used to deal with the interaction between input singles. The number of neurons is not defined. It usually needs to be determined by rules of thumb. The hidden layer usually uses a nonlinear transfer function. In general design, there is no limit to the number of layers in the hidden layer, but one or two layers of hidden layers can handle most of the problems. 3. Output layer: used to represent the output variable of the network. The number of arithmetic units depends on the problem. 201232429 1.2 Activation function The activation function, also known as the transfer function, is a mechanism that simulates the threshold of biological neurons, usually a nonlinear function. The input data and weight value operations are summed up and called the integration function (Summati〇n function). The main function of the activation function is to convert the integration function so that the network can play the role of correcting the weight value. The nature of the activation function affects the function of the neural network to learn nonlinearity, while the nonlinear function avoids the possibility of neurons losing nonlinear characteristics when processing rounded data. These characteristics affect the function of network learning. The common activation functions are as follows: Symmetrical Hard Limit function is shown in Figure 4: -1 if x<0 if 〇:>0 (1.1) 2. Sigmoid function is shown in Figure 5. : =--- fic\ \ 1 + e" (1.2) 3. Hyperbolic tangent function (take 咐 plus ^ Tangent functi〇n) as shown in Figure 6: y(x) = tanh(jc) = ——~~- — /1 〇 \

Ex ^e~x VUJ 4 · Linear function Figure 7: y(v) = fcv (1.4) 1.3 Time-Delay Neural Networks (Time-Delay Neural Networks, hereinafter referred to as TDNN) The multi-layer perceptnm (MLP) 'learning algorithm is also used in the multi-layer feedforward neural network's learning algorithm. The difference from the inverted transmission god 201232429 is that all the influence factors that may affect the current stage of the state are all regarded as the input signals, and the information structure information can be used to express the relationship between the time and the time. The architecture of the inverted-transfer-like neural network, when dealing with timing issues, adds the previous information from the past, transforming the solar phase sequence into a spatial sequence of the Hessian Road, enabling the neural network to learn the time between the information. The related characteristics before and after, namely the time delay type neural network. As shown in Figure 8, the network only has the input data of the pen, but each time it is input, it will contain the first few operations related to the current output of the resource, the front-pen and the first two. Data from different time segments, while turning people to the Internet. By changing the structure of the input data, the network can learn about timing issues. 14 Reward-like neural network Recurrent Neural Netw〇rk (remuneration) is a kind of slogan that can handle the problem (te_ral p job sses). The architecture includes a connection input layer a_tenated input-(10)(10)laye small feedback processing layer (P(10) ssing layer) and an output layer (0utput layer), as shown in FIG. The object TG in the receiving layer represents the memory unit, and the input of the hidden layer is the output of the input layer and the receiving layer. The rest are the same as the inverted transmission type neural network. The so-called _ state refers to the behavior of neurons feeding in and out of the network, and the reverse transmission of Lai Jingqing is different from the static neural network between the occurrence of the fairy neuron or the change of the sinister (four) into (4). The network is closely related to time by returning the output value of the current neurons (Time-Delay). 201232429 Neurons with feedback links in the feedback neural network are called Dynamic neuron. The difference between a feedback neural network and a static inverse neural network is that the input and output of the recursive network are not just mappings, but can be returned. The transmission method keeps the result of the current message processing in the network structure as a reference message for processing the next stage. The characteristics of the feedback neural network are quite suitable for the system that simulates dynamic immediacy. The most important aspect of road plasticity and fast convergence is that you don't need to input all the examples into the network, but you can send the data to the line for instant online learning. Time delay class god. Jing Na and feedback nerve The network has the ability to solve time-related problems. However, the number of delays in the time-delay-like neural network is difficult to determine. If the number of delays is too long, it is easy to cause input. The amount is too large to make the network difficult to train. For example, if there are five items in the input data, the time delay type neural network must have fifteen input neurons. The axis time extension face is clear. However, because the network is relatively large, 'there is a large amount of training data, the training time is long, and the network convergence speed is slow'. The space in practical application is compressed. 1.5 Error learning algorithm error learning algorithm storage forward operation The ratio of the wire to the target value is generally obtained, and the error correction surface is subjected to the (four) direction operation part. In the forward operation process, the data is weighted from the input layer via the hidden layer to obtain the conversion of the integration function through the activation function, and then Pass to the output layer to calculate the output value of the network. # Output can not get the target value, it will carry out the reverse operation 'return the error value, it is expected to modify the weight value of each layer of neurons to make the error reach the target value or tolerable range Through the feedback neural network 201232429, the immediacy adjuster is called online learning method, and the line field method is ugly. There are dynamic and parameter functions. 1.5.1 Forward operation The forward operation is to pass the input value to the weighted product of the weight value to obtain the integrated function and then convert the value into the output value of the processing unit through the activation function, as shown in formula (1.5). (1 6)., Forward unit model. η

Νι=Σ^.·%)+^, ί=, (1.5) yj = f(Vj) = /(^] νν,.Λ:,. + b) fcl (1.6) bj . The weight (bias). π : number of input layer neurons. Wij : Link weighted value. /(_): A conversion function with processing function that converts the sum of the weighted product operations of the input processing unit into a formula for the output value of the processing unit. Usually, it is an exponential function with two-way bending. According to different functions, when the self-variable is close to positive and negative infinity [-0〇, +0〇], the value of the function value will be different. Yes: A signal with a round-out function. 1· 5. 2 Reverse operation “Xi ^ Sense 4 (4) Provincial algorithm] The back-transfer algorithm based on the steepest slope method (Back, Qing ti〇n, βρ) is the most representative. The purpose of learning algorithm It is to reduce the gap between the network output value and the target value. The difference between the output value of the network and the output value of the training target is the network slip difference, which is represented by the error function or the observatory function (Cost funGtlGn), 201232429 The «function minimizes the lightness, and the meaning is as follows (1.7) £=destroy (,η) zj ^The ideal target output value of the jth output unit of the output layer.

Yj . The actual output value of the jth output unit of the output layer. j : number of output layer vectors. The three kinds of processes are called to correct the weight value of the link, so that the error function reaches the most jest, and the error degree is proportional to the partial differential value of the weighted value. And is proportional to the learning rate of 7. Δνν. = -η. M (1.8) 77: The magnitude of the minimum energy function, which controls the stride of each weighted value modification, called the learning rate. w When the network corrects the weight associated with the output layer, the linkage law of the partial differential of the hidden weights of the hidden layer and the kth of the output layer (chain_ _5£_= dEdy, dvk in S^ ^~yk)f\vk) k : the kth neuron of the output layer of the neural network. The amount of error of the kth neuron in the rounded-out layer. J: The jth neuron of the hidden layer of the neural network. The correction amount of the network rounding layer and hidden layer link weighting value is: 201232429 ° When the weight is not related to the network rounding layer, the error function is the first god, the 'star and the hidden layer' The weighting system of meta-weights between meta-divisions is: (^Σ~(rk - y*)· /'(Vjt). wt · /'(v.) · y.

_ f x J δ, ΣΧ jk •/’(V,)

4 · The amount of error of the jth neuron in the hidden layer. When the network is not the output layer, the correction amount of the weighted value % of the connection between the previous layer and the latter layer is: = Ti-^k-yi Similarly, the threshold correction amount of the hidden layer neurons is: Λ/Q - commit J~ η^θΙ = ~η-^ (1.9) 1·6 network parameter class neural network usually contains AMf (four) to leave, the lower touch parameters will affect the learning process results and efficiency: 1. hidden layer number: if not hidden The layer can't construct the nonlinearity between the input and the output in the problem. The excessive number of layers can make the network too complicated and slow down the collection. When the number of layers of the hidden layer is -layer or two layers, there is usually a better convergence and it is sufficient to react to almost all problems. 2_Hidden layer neurons: How many processing units are needed for the hidden layer does not have a certain 201232429 rule. The more the number of hidden layer neurons, the slower the convergence rate, but a smaller error value can be achieved. Too few hidden layer neurons will have a non-linear relationship between the input and output of the problem because the network does not have enough parameters. Excessive number of hidden layer neurons can easily lead to excessive description and learning of the network. 'Let the network learn to the noise part of the input, the number of layers and the health of the hidden layer are usually based on experiments. The law decides, or you can refer to [31], and choose the equations (10) and (1. u) as the calculation method of the number of hidden layer neurons: (1) The number of hidden layer units = (the number of units in the previous layer + The number of units in the next layer is +2. Face (1.10) (2) Number of hidden layer units = (number of input layer units X number of output layer units) 1/2. (1.11) 3. Activation function: The activation function commonly used in reverse-transfer-like neural networks is a double-bend function, while the more commonly used activation function of a dynamic-like neural network is a double-bend tangent function. Different activation functions can be used depending on the nature of the problem and the design of the network. 4. Error function: In addition to the error squared error, the error function can also take the cubic 隹 and even other functions to express the difference between the network output and the target output value, and improve the learning quality. 5. Learning Rate: The learning rate is too large or too small, which is detrimental to the convergence nature of the network. Although the larger learning rate has a more obvious network weighting correction amount, it can approach the minimum value of the error function faster, but it is easy to cause the error to oscillate. If the learning rate is too small, the convergence speed will be too slow. Therefore, the learning rate can be changed. In the initial stage of the network, a larger learning rate can be adopted. As the network training progresses, the party 12 is reduced. - The appropriate learning rate is between 0.6 and Q 8. If the learning rate of the change is taken, then the learning rate is multiplied by the coefficient of __ less than 1G, and the learning rate is gradually reduced, but the lower limit value cannot be less than this value. However, the appropriate learning rate for some problems may be as low as 〇·1 or less. 6. Inertia (Momentuin): The addition of inertia is mainly to bring the update of each generation weight to the direction of the pre-generation, so as to improve the seismic recording of the learning materials and to enlarge the search direction for the smooth target value. Achieve convergence effect. If the inertia factor is too large or too small, the convergence of the network is not good, so the inertia factor can be adjusted in the same way as the learning rate. • Batch '"Batch Learning: The standard inverse-transfer-like neural network learning algorithm is to update the weight value every time a set of training examples is loaded, called "case-by-case learning." In batch learning, after loading all the training samples, the correction amount of each stroke is added and processed to update the weight change. Random Perturbation: The learning of the neural network is to minimize the error, but the steepest slope method has a situation where it will fall into the local minimum (l〇cal minimum), so that the convergence result is not perfect. In order to solve this problem, the ability to search for the solution of the trip region can be increased by combining the Cauchy machine, such as the formula (丨 12). Wn+1 =wn +Δνν+Ω (1 η) Ω : random disturbance amount, between [_〇.〇〇1, +0.001]. 8. Weight initialization: The random number of the initial value of the weight between [-2-1, +2-1] has a good convergence effect. When the weight range is in [-ρ,Ρ], the weight range of the neural network is set to [-d,d], and the formula of d is as follows (1.13). 13 201232429 2P (1.13) π : The number of input layer inputs. P: Weight range, which needs to be an integer. 9. Normalization (N〇rmaiize is wide because different activation functions have different ranges of values. If the input value is too large, the neuron output will be the saturation value of the activation function, and the difference in the data cannot be felt. Perfect learning, so the input and output can be orthodox with different Wei functions, so that the nucleus is within the range of activation and softness. 10. Processing unit saturation: The activation function is either using the double bending function or the double ridge function. When the self-variables approach a number equivalent to A, the function differential value will approach zero, making the network unable to combine the effect of the weighting value. To avoid the problem of processing unit saturation, add a value to the activation function differential value. The invention realizes the double-bending function and the double-bend tangent function hard-like structure of the neural network based on the segment linear method, and improves the duck-like nature, and the Jingxing (4) is matched with the high-speed pipeline design method and the segmentation calculation. Miscellaneous, the hardware array part is shouted and shared, and the design can let the individual change the network hardware architecture with the problem, replacing the software that requires lengthy calculation. Body Architecture and Principles 2.1.1 Forward Hardware Architecture The architecture of the reverse-transfer-like neural network and the feedback-based neural network are all multi-layer perceptron architectures. Each-layer operation has a hierarchy of layers before and after, and if it is hard. If the body is designed with the number of layers, then the components will increase with the secret structure, and the cost will increase accordingly. 201232429 Due to the complication of the neural network, the back-layer will be fresh and smooth—the value of the layer It can be operated. Even if each test is designed to have an independent reciprocal unit, the increase of the overall operation speed is based on the doubling of the component. Therefore, the data of the pre-layer of the neural network can be passed through the round-trip convergence. The transfer is performed, and the result of the operation is stored by the memory. When the operation is performed in the lower layer operation, the stored value is placed on the input bus to reduce the amount of hardware used. The present invention is based on the prior art ring serial architecture. As shown in Figure 11. This architecture can reduce the number of logic elements in the domain. The operation is based on many operations, = the number of operations, the more computing units can be calculated, but the number of computing units will often Correcting cost and hardware _, reducing the possibility of practical application. Therefore, the present invention is also a method of segmentation calculation. ### The road hardware is too much or the cost is too large to be collateralized. The number of side logic components is fixed and the capacity of the memory is used to complete the large ugly neural network operation with a limited number of gods. After the hardware network is initialized, the input layer data is transmitted from the input bus to each Operation = Intra- and 'Operation Units' (4) "The value of the operation of the surface function is transferred to the hardware block of the activation function, and the calculation of the activation function and its differential is completed. After the operation is completed, the song will shift the signal (shi(1) When enabled, the values calculated by the respective arithmetic units are transferred to the external memory of the arithmetic unit for storage, and then the result of the calculation of the surface of the function is completed, and the input of the level is performed. Phase: Action. Repeatedly making this architecture, you can complete the three-layer, four-layer class of God's network. Figure 12 is the forward computing architecture diagram. 15 201232429 Since the number of logic elements of the actual hardware in the above-mentioned ring-and-column architecture affects the size of the established network, the present invention implements the entire generation of the gods. Figure 13 shows the flow of the segmentation calculation. It is assumed that the maximum number of processing units that can be synthesized is 2, but the number of _, _ roads that need to be calculated is 5, and the operation mode is In order to complete the calculation of the former _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ , δ blocks of hidden layers, calculation of hardware blocks such as blocks and update weights. The forward calculation divides the calculated activation hearing and differential results into difficult words. In the error calculation of the hidden layer, since the operation process is a multiplication and addition operation, the concept of the forward operation is the same as that of the hardware block without the activation function, so the present invention adopts the hardware architecture of the forward operation and parallel. The concept of processing 'removes the part of the activation function in the forward direction to complete the hardware of the hidden layer error gauge. When the reverse operation starts, the target value and the value of the output layer obtained by the forward operation and its differential value are first transferred to the 5 blocks of the calculation output layer. In this block, after the target value is subtracted from the calculation result of the output layer and multiplied by the differential value, the gap of the output layer can be obtained. After the calculation of the gap of the δ 〇 output layer, the layer gap will continue to be hidden. The amount of operations. Since the operation of the hidden layer must be multiplied by the weight of the processing layer and the weight of the processing unit, the operation is performed after the multiplication and accumulation operation is performed through the architecture of the circular string, and the result is combined with the hidden layer nerve in the forward operation. The differential value of the element is transmitted to the δ block of the hidden layer, and the block can obtain the gap of the working layer only after multiplying the value obtained by multiplying the two by 201232429. If 疋, the road is a four-layer reverse transfer-like neural network, because the error amount is calculated in the same way. 7 The same architecture and method are used to complete the god between the second hidden layer and the hidden layer of the first layer. The logic component is then used to build the hardware block of the layer. After the calculation of the difference amount of each arithmetic unit is completed, the correction amount of each weight value is calculated by the block of ~~. The amount of gap between the output layer and the hidden layer is stored in the same -1 memory. Since the gap of the output layer is the first to be received, the output layer peach will be read first, so the output layer and the hidden layer will also be calculated first. For example, the input of the block contains the output value of the delta memory, the learning f rate, and the individual neurons in the architecture of the input neural network, and the collapse flow in the object block will be When reading the value of the delta memory towel, the concept of the bias value is transmitted to the wheeled busbar such as the block, and then the front-layer_input is transmitted to the input block of the block to calculate each nerve. Yuan partial rights pavilion and Wei Jia miscellaneous _ positive amount. On the same day, the matching of the nuclear matching timing passes the weight value of the pre-generation data and the value of w to the hardware block of the updated weight, and the weight value of the pre-generation will be more affected by the value of Aw in the dumping block. Completion—the reverse operation, and Figure 14 is the reverse operation architecture diagram. 2.1.3 Arithmetic Unit Both the forward operation and the reverse operation of the present invention use a ring-shaped hierarchical structure, and the ring-like series is a result of concatenating the results of the respective arithmetic units. The internal structure of the arithmetic unit is mainly composed of four parts: memory, multiply accumulator, shift register and activation function. These four parts are independent hardware blocks, and the high-speed pipeline design method is used to disassemble the operation into independent operation, which can greatly improve the performance. 17 201232429 Inside the nose unit. The P memory contains the weight value required for the calculation, and the neural network must perform a weighted operation for each __ input value. Therefore, it must be designed inside the operation unit. Multiply accumulator. The forward and reverse operations of _God_Route contain the notion of threat, which is very important for the dragon sequence. In order to simplify the design of the controller and read the corresponding weights in the correct order, the arithmetic unit The internal design has a memory of the - (four) (touch) architecture, and only needs to control the order of writing. Then the sequence of readings will be "with." There is an additional addressing method in each unit for converging the weights. Row (four) is written to the timing diagram of each arithmetic unit.

The face is the address of the memory of the computing unit in the computing unit, and the male is taken from the city because it processes the data in parallel. Since the frame of the ring string is formed by connecting the arithmetic units in series, each arithmetic unit calculates the cumulative operation 岐 simultaneous axis. In the forward-looking operation, the neural network must convert the input message through the activation function. In the name design, each operation unit represents _ neurons, so the computational materials each have independent functionalization blocks. The activation function hard pet block is mainly composed of two large blocks, including the double "function and its differential block and the double f-curve function and its differential block. The data is transmitted through the input and flow transfer accumulators. In the arithmetic unit, the memory is added to the accumulated value of _, and the value is transferred to the two activation functions _ block for operation. Each operation single handle has a control signal for shifting, and this signal controls whether or not the current material will be transported. Yuan Shisi loses (4) under - Jing material. «Material input and after the operation of the single 兀MM, the sub-parallel architecture in each surface calculation unit, through the shift method, the data is transferred to the next operation unit. Pin 5 is the arithmetic unit signal 18 201232429 2.1. 4 stacking, arranging and random access memory memory transfer neural network and feedback neural network in the process of operation need to access data for calculation, and feedback The feedback structure of the neural network must also transfer the data stored in the memory to the arithmetic unit to perform the feedback to achieve the feedback behavior. Therefore, the memory configuration is very important. The memory access mode of the system is generally matched with the memory. The physical address is stored or read, but in order to simplify the design of the controller and the order in which the data is stored, this paper is divided into three different types: weight data (Stad〇, side (FIF (1), and random access memory). The hard and thin structure is stored. The functions of the stack and the hardware structure signal are as follows: (1) clear · In the stack architecture, the write position is returned to the lowest position, and the read position is returned to the most position; In the architecture, both the write and read positions are returned to the lowest position. (2) hold: indicates the current memory location in the stack and queue structure. (3) restart: This signal needs to be matched with the h〇id signal. The stacking and array architecture indicates that the reading position is returned to the position set by the previous hold. β (4) rd_addr: memory address table for reading data. ι memory hardware architecture data content name | Weight_RAM Storage weight 4 LearningAddr_FIFO Stores the inverse ^ 辱 权 weight value ϋ memory location UpgradeAddr_FIFO Store 丨 i positive weight - weight; out memory location 鲞 target_RAM ^ memory targetAd dr_FIFO stores the memory location of the target value output input_FIFO stores the input data PE_Weight-FIFO stores the weight within a single neuron ' ----....丨.. 201232429

Y FIFO dy_Stack

Recurrent-FIFO Y Stack ^iM^ib4* 丨向正$算过纯结的结 f 丨„_^ι路咖的 ΐ ΐ 结果 所 所 i i

Delta FIFO new_Weight_Stack

Result FIFO ί out of layer 5 and hidden multi-distribution *------------- The weight of the training is 1.1.1. The numerical principle of the digital system is more represented by the binary digital system. Once and the processing is also necessary to 剌 binary signal of the second check component to the actual ^ _ _ with 32 of the binary code timing number money as the _ numerical coding method ' and N1 〇 sn micro-processing is _ view The 754 format coffee decimal, so in the next touch and __ _ _, face peach to the fixed point converter to enter the value. In the domain ten, the number of digits and the number of decimal places can be changed according to the user. 3.1 Activation Function Hardware Architecture 3.1.1 Activation Function Characteristics The architecture has two fine activation functions, which are f-synchronous two-phase tangent functions. Both the double bending function and the double engine tangent function have symmetrical properties as shown in Fig. 17, eighteen. Therefore, after the hardware design is used to obtain the right half function, the left half of the function is obtained according to the formula (1 14) and the eight 4 μ ις, + 〇) and the formula (1.15) using the subtractor and the multiplexer's judgment. The way to reduce the near-half of the display component and Figure twenty.口1920 201232429 00 = j / (λ) , for x > 〇 IWO) , forx < 〇 (1.14) y (x) = · [ / ( text ) , for x > 〇 1 ~ / ( λ ), f〇rx<〇(1.15) 3.1.2 Fragment Linear Method# The linear method is to calculate the approximate double bending function curve using the Center Linear Approximation (CRI) algorithm, as shown in Table 2. Where x in the table represents the input value, q generation

The number of people in the table represents the depth of the interpolation (Interp〇lati〇n 〇 printed). After testing, the call is 2 hours, s is set to 0. 28094 has the best solution. The rest of the algorithm can be calculated using symmetrical properties to reduce the number of logical components used. The hardware architecture is shown in Figure 21. Figure 22 is a schematic diagram of the segment linear method. It can be seen from the figure that the segment linear method is a non-linear two-material function in the way of __method. The principle is that after finding the turning point, the error will increase slowly. When the #error value reaches the upper limit, a new turning point will be found. This hardware _ requires a total of 817 logic ploughs and has to take a clock to get the result.

The fragment linear method virtual code is as follows: g(-c) = y, U) = 〇' /iU) = >^2(x) = 1(1 + -^) for(i = 0; i = q;i + +) { g'(x) = max[g(x),h(x)] Κχ) = ^(8(χ) + Κχ) + ά,) g(x) = g\x) 21 201232429 3.1.3 Fragment Linear Method Architecture In the case of modifying the q value to 2, the repeatability of the hardware block is utilized to simplify the hardware architecture. The fragment linear algorithm in Table 3.2 requires an adder and two shift registers using the formula (^6). After changing the formula (1.17), it only takes one shift register and one adder, and the result is calculated by one clock less than the former. In the first operation where q is ,, the formula (1.18) requires two adders and one shift register. Since the initial value of g(x) is 〇, the formula (3·17) is substituted. The formula (119> formula (1 19) not only costs -addition H and -shift temporary storage ^, and can combine the variable values. Table 3. The structure of 2 can be increased by adjusting the number of iterations q Degree, however, the number of iterations must be more and more "and also need more (four) transport between the two. The results of the job found that the iterations are relatively flat _ method. The hair of the dynasty dynasty: the seam "2, the formula Α2〇) changed to - the temporary copper of the storage constant. Not only can the logic elements and objects required by the county type be reduced. (4) In the case of the coffee gate, the formula (1·18) of the first iteration is changed to the formula (ι). 21), you can use this constant to increase the accuracy, as shown in Figure 20. This hardware architecture requires only 333 logic elements, and 8 clocks can be used to get the result. ) (1.16)

X

h(X) = ~2\ (1.17) Λω = -(^(χ) + Λ(Λ) + Δ) (1.18) 22 201232429 Λ(λ) = ^(〇+ - + - + Λ) 2 2 4 ' 1 λ: Δ =—+—+— 4 8 2 (κ)+ battalion

X △, + 4 Λ W = - (g (jc) + Λ(χ) + Δ 2)

3·1.4 Double Bending Tangent Function Architecture (1-19)(1.20)(1.21)

The double-bend tangent function differs from the graph of the double-bend function in the range of the range between the two. A—14 is the graph of g(1) and h(x) in the case of the # segment linear method in the iteration. It can be seen from Fig. 15 that the red line represents g(x) and h(1) is the shape drawn by the larger one. It can be seen from the graph that the 'fragment linear method is the size of the constant in the non-linear it diagram liter/hardwood structure interleaved by interpolation and the influence of the shift register on the triangle in Fig. 24. It also affects the curve function graph synthesized by the segment linear method. The value of the fUl register and the shift register are used to implement the double f-curve function because the /_deletion-segment is completed. In addition, the value of the tangent function is in H, and the center of symmetry is 0. The male represents the center of symmetry of the y-axis of the double-bend function. Through the empirical method, the 0.5-hard-bending double-bend tangent function block, as shown in Fig. 20 “D” after the number of real logic elements, requires 9 _ pulses to simplify 纟U. This hardware _ total 3 Π 片段 1.5 segment linear method related parameters 23 201232429 The variation Δ of the segment linear method represents the depth of the internal difference in the algorithm. Shifting the temporary storage $ for each iteration of the iteration will reduce the value by grading the Δ;& However, in the first section, the segment linear method architecture is modified when q is 2, and according to the slope of the secret portion of the line segment in the algorithm architecture, the modified V 4 and the shift of the segment linear method architecture are adjusted. Temporary - make the activation function more precise. Figure 27 shows the implementation of the fragment linear method in the prior art, and Guan 28 is the completed figure after the architecture modification. In the modified architecture of Figure-19, tf', the error is small in the case where the two iterations have their own independent * and ~. Figure 3: The double-curve tangent function _, which is the parameter of the miscellaneous money table 3 2 modified by the fragmentary secret method, is shown in Figure 30. Table 3.2 Fragment linear method parameters and results comparison Network architecture double f-curve function modified double-curve function modified double-bend tangent function q 2 2 2 0.476807 Λ 0.28094 0.979721 0.070235 0.197876 0.188034 Maximum error 0.024458 〇. 020027 0.051239 Required logic components 817 LE 333 LE 317 LE operation time 12 elk 8 elk 9 elk 3.2 control unit control single master the entire system operation 'master the entire system execution order. It is mainly divided into four parts, which include the control of the weight order, the control of the operation unit address, the control of the target value, and the control of the overall m-range. Control II's design approach is based on a finite state machine design pattern. In addition to controlling the segmentation operation, the controller of the overall system also controls the activation function block, the accumulation n, the parallel learning architecture, the output layer error calculation, and the hidden layer error calculation in each operation unit. 24 201232429 3·2. Weight Memory Management Unit The class neural network is trained in the whole time. The order of the weight values is used in three different orders according to the steps taken by the network. The first is the order in which the weight values are used in the forward operation, the second is the order of the weight values in the reverse operation, and the third is in the order of updating the weight values. Therefore, the controller of the present invention is designed to store two different order weight addresses in the memory. The memory that stores the weight value will get a different order according to the ship that reads the different storage addresses according to the situation. The right-handling unit is based on the order of the three __, the address of the control clerk's memory, which changes the order of the weight value output, and simplifies the complexity of the system. If the network architecture is a four-layer inverted-transition-like neural network, the number of input layer neurons is two, the number of layers in the first-layer hidden layer is two, and the number of neurons in the second layer is three, and the output layer The number of neurons is four, and two of the green domain units that can be combined with the hardware are shown in FIG. Since the ring-and-column architecture has a parallel characteristic relationship, when the controller reads the instructions read under the memory within 7L, all the arithmetic units will simultaneously transfer the first -> in the memory. As shown in Figure 32 and Figure 33, when the neurons in the input layer transmit the data to the ring structure, they will simultaneously calculate the w01 in the memory unit and the memory in the memory unit. 122 reads out and operates with the input. When the data of the hidden layer is calculated by the hardware block of the unified function, the two neurons of the hidden layer are then input. At this time, since there are only two processing units, the segmentation calculation must be performed. Therefore, wl3, w23, w33 in the arithmetic unit-memory and wl4, w24, and w44 in the arithmetic unit two memory are read out and operated. Then repeatedly read the output value of the first hidden layer, and read the wl5, w25, and which in the arithmetic unit-memory, since the operation signal 25 201232429 yuan reads the same signal, it must be as shown in Figure 33, Figure 30 As shown in FIG. 4, the memory of the arithmetic unit 2 must have the same number of weight values as the arithmetic unit, so that the W37, W47, and W57 behind the second memory of the arithmetic unit are not read, resulting in disorder of the overall system. The first sequence uses the characteristics of the queue memory to output the weights in order. When the network starts to carry out the hardware primitive, the weight control sequence starts to work, and the address sequence of the second and third cases is sequentially written into the queue memory. The second order is the case in the inverse operation of the neural network. The hidden layer error of the inverse operation is computed through the architecture of the ring string. At this point, the output layer can be regarded as the input layer of the instrument—the layer hidden layer is regarded as the first layer of hidden layer, that is, the entire network is reversed. The weight value must be in the order of 8, Place the memory in the operation unit -quot, then place w49, w48, w4?, _ in the order of the operation order. The weight value has a corresponding storage address according to the input order of the network initially as shown in Fig. 24. Therefore, the weight value is greeted by the memory address 25 21, 17 5 as the second order. Since the reverse operation will adopt the architecture of the clothing string J, as the arithmetic unit may read the weight value required for the next layer operation, it must also be considered as shown in Figure 33 in each arithmetic unit. The memory is filled with the case where the correlation weights are required for each layer calculation. The first-type If% reading is based on the update weight value. Since the update method of the reverse transfer algorithm is to update from the weight of the last level of the network architecture, and update the weight value "to __ shape (four) pro, (4) consider the operation unit record (four) to fill the same two times 6 In the case of the weighting of the weights, the memory addresses n 2 11 . are sequentially stored in the order of the Suining, so that the weight values are read in the reverse order of 26 201232429 as the third order. The following 疋 weights S memory management unit important signal description: 1 · ULFIFO-clear: This message shows the memory reading and writing address of the second-order order of the age back to the lowest position. 2. UA-FIF0_clear: This signal indicates that the third-order memory read and write address will be returned to the lowest value. 3 Initialize: This signal indicates that when all network parameters are initialized, they will be based on Na. In the case of flipping, sizing, and the most local unit, start computing the address of the management weight placement order. 4. Done: This signal will generate a high potential when the weight management unit is initialized, which is convenient for the controller to control. .Rd_opcode. When the signal is 〇〇, the order of the right value is output from the memory, and the second order is "01". The third order when updating weights. The weight management sheet is the program fragment when it is processed in the second case as shown in Table 3. 3, and the calculations of Zhong Jianjian are in the same layer, but each slave is based on the front of each layer of the network (4) (4), and the F (4) anUayei_jQde, _____ ,

Next—Layer—McxUx and Address's Register' can calculate the order of the addresses required for each layer of weight memory. -~~---Table 3·3 weight management unit performs the second sequence program fragment

If ( neural_netowrk_kind=Recurrent) then _ Flrst-Layer_Node=Input_Node+Hidden l_Layer_Node 27 201232429

Else First-Layer-Node = Input-Node End if /* If the network is a four-layer network, compare the number of neurons in the second hidden layer*/ MaxLayerNode=MAX[Input_Layer_Node, Hidden-Layer-Node, Output A Layer-Node] if (MaxLayerNode>MaxPE) then /*MaxPE=The maximum number of hardware can be synthesized MaxNode=MaxPE else MaxNode^MaxLayerNode end if the quotient and remainder of each layer of neurons divided by MaxNode*/ if ( RAM/=0) then Mod=Mod+l end if Forward_Layer-Node=Output-Layer-Node Next_Layer_Node=Hidden_Layer_Node Next_Layer_ Mod= Hidden_Layer_Mod Address=Max_Memory-Address-1 Original-Address=Address Total_Counter= Fonvard_Layer_Node*MaxNode*Next-_Layer_Mod Used_Counter= Forward_Layer_Node* Next_Layer_Node Counter= Next—Layer—Node If (TotaLCounter=0) then Return Elsif (Used-Counter =0) then Save Address

28 201232429

Total-CounterTM

Elsif (Counter=0) then Save Address

Address: Original-Address-1 Original-Address—

Used Counter-Total _Counter—

Else

Save Address

Address= Address- Forward_Layer_Node

Counter-

Used_Counter—

Total one Counter—

End if 3.2.2 Address Management Unit The circular serial array architecture is composed of a number of arithmetic units connected in series, each of which is a neuron. Each arithmetic unit has its own independent address. Since the Quanlong flow row and the input bus bar are connected to each operation unit, when the weight value on the weight bus bar is to be written into an operation unit, the weight value is determined according to the address on the address bus bar. Enter the memory of the specified arithmetic unit. The controller of the address of the single-layered computing unit is used to control the number of the 7L address of the computing unit according to the type of the network and the size of the maximum single-combination operation, thereby reducing the burden on the system controller. When the network is operating in a ring string, the type of the data network and the neurons in each layer of the network sigh the same reading material, and the seam value will be placed in different processing units. To write full and wll to the memory of the arithmetic unit, it must be as shown in Figure 35, 201232429. When the weight is full, wll is written to the weight bus and the signal on the address bus is 1, then the person will be read into the - (4) calculation unit. Then, the weights w02 and w22 are written to the weight bus and the address bus is adjusted to 2, as shown in Figure 36. All weight values are arranged in accordance with Figure 37, and with the weight controller of the previous section, all weight values can be stored in the correct arithmetic unit. In order to simplify the controller, the present invention separates the controllers of the address busbars independently. The program fragment of the address management unit is solved in Table 3.4, and the calculation of the address of each layer of the operation unit is the same, but each operation is modified according to the context of the network layers.

Forward-Layer_Node, Next-Layer-Node, and Next_Layer__ Mod's scratchpad' can calculate the order of the address of each layer of arithmetic unit. Table 3.4 Address Management Unit Program Fragment If (neural_netowrk_type=Recurrent) then

First_Layer_Node=Input_Node+Hiddenl_Layer_Node

Else

First_Layer_Node = Input_Node End if /* If the network is a four-layer network, compare the number of neurons in the second hidden layer*/ MaxLayerNode=MAX[Input_Layer_Node,

Hidden_Layer_Node,

Output_Layer_Node] if (MaxLayerNode>MaxPE) then /*MaxPE=The maximum number of hardware can be synthesized~

MaxNode=MaxPE

MaxNode=MaxLayerNode end if 201232429 /* The quotient and remainder of each layer of neurons divided by MaxNode if (RAM/=0) then Mod=Mod+l end if

Forward-Layer-Node=Output_Layer-Node Next_Layer_Node=Hidden_Layer_Node Next_Layer_ Mod= Hidden_Layer_Mod Address= 1

Total_Counter=( Forward-Layer-Node+1)* MaxNode* Next—Layer_ Mod Counter= Forward-Layer-Node If (Total_Cpunter=0) then Return

Elsif (Counter=0 AND Address= MaxNode) then Save Address

Counter= Forward_Layer_Node Address: 1 Total—Counter—

Elsif (Counter=0) then Save Address

Counter: Forward—Layer-Node Address= Address+1 Total—Counter—

Else

Save Address Counter—

Total_Counter—3· 2· 3 Target value management unit Because of the reverse operation, the inverse transfer learning algorithm starts from the last god of the turn-out layer. If the chaotic target value causes the bribe (4)_ to be stored, the input value must be entered from the target value of the last neuron in the output layer. Since the target value is stored in the memory, when the network #multiple data needs to be trained, it must be input from the last data of the first group into the front memory, and the input of the training data is from the first - The first - of the group, in this way, the target value is opposite to the input order of the network training material. In order to solve such a situation, the controller of the present invention is designed to input a target value only in the order of the first group of the first neuron and the second group of the first neuron, and the controller will It is judged how to write to the queue memory of the target value. If the network output layer has four neurons and there are three sets of training data, the target value input sequence is shown in Figure 26. According to the inverse transfer learning algorithm, the target value will be calculated from the tl4 stored in the last address 3 in the first group, followed by the address 2, the address 丄, and finally the target value of the address 0. After the first set of training is completed, the training of the second group is started by switching to t24 stored in the last address 7 of the second group. Figure 27 shows the target value algorithm. First, the total number of input values of the target value is stored in the scratchpad. Then the write group starts from the first group, and the number of neurons written is also from the first one. Calculated. Write the memory address of the last neuron stored in the first set of data to WriteDa1;a. When the addresses of all the neurons in the first group are written, the write group is changed to the next-group until All addresses have been written to the queue memory from the group to the front. The following is the virtual code of the target value management unit: _ write group = 1 write neurons number = if (total number of input = 〇) then jump out 32 201232429 elsif (write neurons number = output layer nerve The number of elements) then changes the group number to be written to the number of neurons to be written in the next group from the first to count the number of elements (group * output layer neurons - number of neurons) Write end if 3.2.4 System Control Unit The entire class of neural system operations is controlled by the system's control unit. All the values are left in memory, and (4) H's goal is to write or read the data in the memory in conjunction with the timing. The design method of controlling n is based on the finite state machine design pattern shown in Figure 19. In addition to accessing stored memory, the controller also controls the entire %-like array architecture, segmentation calculations, accumulator n-signal selection lines, and control units that control the first three bars. The main flow and state of the controller are shown in Figure 40. The flow chart is divided into three major blocks, where the initial signal is generated, and the weight, health and target value controllers are calculated according to the type and size of the network after receiving the signal. Waiting for the system to accept the weight and other such; the completion of the I* unit, the control of the weight of the management unit, etc., the data is sequentially placed in the queue memory to complete the preparation of the system. The part of the forward calculation begins when the system is ready. First, the controller adjusts the signal selection line of the worker according to the type of the first-hidden layer activation function. The training data is then transferred from the input array memory to the ring architecture for operation unit array operations. If it is a feedback-type god _ after the gift is written, the dragon of the receiving layer is also transmitted to the ring structure, and the value of the first-hidden layer neuron is calculated, and then the size of the network 33 201232429 is determined. The segment turns, the end of the silk - layer _ operation. Adjust the signal selection line of the multiplexer according to the number of layers of the hidden age. The operation of the hidden layer and the round-out layer is to store the value of the memory command in the calculation data of the front-layer profit-seeking financial (4) block, and then return it to the ring-like serial structure to complete the whole forward operation. - After the system performs the forward calculation, it will decide the subsequent actions according to whether it is the training network or the test network. In the case of network training, the controller will continue to perform the reverse operation; if it is for the network, the value produced by the output layer will be transmitted to the output data of the result. At the start of the reverse operation, the value of the hidden value of the value obtained by the control/target value control unit and the value obtained by the forward operation and output layer are then transmitted through the structure of the ring string and the hidden layer error amount_block. Lai layer county. After the countdown of the county is completed, the controller will calculate the amount of error, the rate of learning f, and the input value of each pre-neuronal level, and the weight of the weight management sheet will be used to calculate the new weight. Finally, after the touch is completed, and the number of iterations set by _ is completed, the control of the entire controller is completed. The %-like string (4) touches the block of the hidden layer error amount to obtain the _ layer layer error amount. _ After the error amount is calculated, the amount of error, the learning rate, and the previous stage of each neuron are controlled, and the time of the heavy tube element is matched with the time of Awaki County, and the group_weight is obtained. Finally, whether the mine training data is completed and the number of iterations reached is achieved, and the control of the entire controller is completed. 3·3 software planning The software planning aspect of the system is to send the command to the bus bar via the microprocessor. The Avalon bus is designed according to the temporary storage block, and the hardware commands are issued. For example, the type of the road is changed. Size, activation function, series: fine, slit, target value, etc. The advantage of this design is that you can set the 'no' reset hardware and _ directly on the software interface when you want to modify the type and size of the network. _10—The architecture of the system and Nios „ used by the present invention. The decimal number used by the software is a floating point fraction in the lion 54 format due to the hardware calculation of the decimal point. Chen _ 妓, (4) material-like converter placement: on the hardware end. Finally, when the hardware operation is completed, the fixed-point address content of the stored data is converted in the format of a fractional decimal number, and the binary characteristic is shifted to the left of the decimal place. The number of elements, = the number of foreign points converted to decimal, so that the converter of fixed-point conversion floating point can be reduced, and the clock of the system is improved. Figure 42 shows the flow chart of the software. The normal operation of the system is in addition to the hardware design. It must be done by software. First of all, it must be given a hard weight (10) age by the code, and then the activation function of each layer of the network _, network architecture, network testing or training, weighting materials, target tribute, input data 'learning' The rate, the number of times the word is looped, and when the network is set and she edits the software for the age of the interface, the instruction to read the result can be executed. The table U is the hardware of the controlled end and the software of the host. . ^ Face communication signal picture menu "rst: == - Learning_Rate

Number of trainings ^00 : Ming..., _τ.丨—......_........丨...

Set all ϋϋΐϋ write and ' start

Epoch

Kind 35 201232429 0x01 . Four-layer inverted transfer neural network & | 0x10: standard feedback neural network 0χΐΐ: feedback neural network and feedback j weight value will be corrected; Af.Hiddenl 5 write vector first layer conceal activation 12^~~11~2nd layer~- Af_Hidden2 6 Write vector Af-Output 7 Write vector round-trip activation function ~~~---; Input_num 8 Write integer input layer #经元一一—! Hiddenl_num 9 Write Integer Γ first 7 layers hidden layer god Hidden2_num 10 write integer second layer hidden layer neurons ready~~~~ ̄ 层层 neurons ^ ~^ Output_num 11 Write integer Train_or one Test 12 Write bit 〇: Network is Training shape. i ---- 1: Network is the test state Weight 13 Write the fixed point - θ input weight value data ~~~~~---- Target 14 Write fixed point - write the target - - Input.data 15 Write - ........................................ done 0 read bit -- - ̄ After the weight value / Result T" 1 - Fixed point _ When the network is in the test state, the output is the output result of the output layer 1 [Embodiment] The present invention calls the three-layer and four-layer inverted transfer gods The sinusoidal curve fitting is performed via the network, two different feedback neural networks are used to predict the residual battery power, and the results and errors of the hardware training and the segmentation calculation are not used. In the training time, the command of Ni〇s II is used to make the network start training to compare the total cost of the return training completion instruction with MATLAB. The Windows XP SP3 ' CPU for this test environment operating system is Intel CoreTM2 Duo Processor E8400 (6M Cache, 3. 00 GHz, 1333 MHz FSB), and the system memory is DDRII 2GB. 36 201232429 The hardware uses the Quartus II 9.0 (32-Bit) and Nios II 9. 0 IDE, and the signal simulation uses the ModelSim-Altera 6.4a (Quartus II 9.0) Starter Edition. The training sample error and the test sample error defined by the present invention are calculated by the Root Mean Square Error method, as in Equation (4.1). RMSE =

(Destinatio η - Output)2 V OutputNode Development System (4.1)

HI development software The present invention uses a Very High Speed Integrated Circuit Hardware Description Language (VHDL). 11.2 Development of hardware components using a Field Programmable Gate Array (FPGA), and Altera's FPGA development platform Stratix II EP2S60F1020C4 is hardware developed. The system's working clock is 100MHz, and the Aval〇n bus is connected to the NIOS II embedded processor and user logic (User L〇gic) and other peripheral (I/O) for system verification, ie experimental data analysis. 4.1.3 Nios II Embedded Processor

The Nios II embedded processor is a second-generation processor developed by Altera Corporation, which is a Soft-Core 32Reduced Instruction Set Computing (RISC) processor. Through the s〇pc (system On a Programming Chip) Builder development system in the Quartus software, the internal logic component (Logic Element, LE) of the FPGA is combined with the Ni〇s ιι processor generated by the user, with the user's own design of 201232429 hard. The body element, the memory unit, the device interface, and various ips, etc., are synthesized into a system on chip system and can be burned into a programmable chip. Ni〇s π is based on Eclipse's Nios II Integrated Development Environment (Integrated Develpence Environment IDE), supports custom instructions (Cust〇m instructi〇n) and has separate programs and data buss for maximum flexibility Users can choose among a variety of system settings to achieve the goal of balancing performance and cost. The Nios II of the present invention is responsible for setting relevant parameters of the neural network and training samples. The operation process is performed by the hardware controller. The Nival II built-in Avalon bus is used to connect the hardware and software. The Nios II hardware architecture is shown in Figure 43. 4.1.4 Avalon Busbar

The Nios II uses the Avalon bus. The main function is to connect the system processor to the peripheral interface. It describes the connection between the master-slave architecture and the timing relationship between the components. As shown in Figure 44. The Avalon bus has the following features: 1. Clock synchronization: The signals on the Avalon Bus are synchronized with the Avalon clock. The advantage is that it simplifies the timing control 'Handshaking and Acknowledge mechanism' Therefore, it is possible to avoid the limitation of timing and facilitate the transmission of the high-speed device. 2. Separation of signals: The control signals, data signals and position signals on the Avalon Bus use separate communication ports, which are characterized by simplifying the design of the transmission interface. 3_Dynamic Width: On the Avalon Bus, the Master and Slave can transmit 8, 16, and 32 bits of data, which is characterized by considerable flexibility in design. 4. Arbitration technology: On the Avalon Bus, the master and the slave can simultaneously make 38 201232429 - use bus to transmit 'characteristics to reduce the bandwidth limit and can handle multiple signals. 5. Memory-mapped: The controlled end can be controlled on the Avalon Bus by means of a sensible body map, which is characterized by the fact that the Chipseiect signal can ignore all external signals.

The write and read signals of the controlled end on the Avalon Bus are shown in Figure 45 and Figure 46. Chipseiect is used to enable the controlled end of the access. Since Avalon uses the memory mapping method to access the controlled data, the Write signal and the Read signal are used to control the reading of the data. Outside of writing, still have to pass

Address Bus access to the internal data of the controlled terminal. 4.2 Experimental Results 4. 2·1 Curve Fitting In order to verify whether the system architecture proposed by the present invention is correct and the accuracy is accurate, the experiment uses the sine function curve fitting problem to carry out the experiment. The experimental parameters are shown in Table 4.1. In this experiment, we use the network architecture of smashing to train the problem of changing a sinusoidal function. The equation is (4.1). The weight of this experiment is I5. The number of random numbers is [-0·5, 0.5]. The average error is shown in Figure 47~ Figure 49. The blue in Figure 47 represents the same weight. The training result 'green' represents the software training result, and red is the actual curve. The results of the analysis are shown in Table 4.2. j(jc) = —(sin(j:) + l) (4.1) Class 4.i. Sine function curve fitting related 兹 兹 —— —— —— rakw1 from · η circle; two ... ..... Ί Ί Ί Parameter working clock ί !_ 100MHz 39 201232429 Integer number 16 bits ~~~-·| ———- .— ----' - ---- . ___ u Decimal Number of digits 16 bits _ ~1 Network type Three layers of inverted neural network ί Input number of neurons * .........**------------ ··------------ 1 1 , ------------- 1 The number of hidden layer neurons is 5 - 1 The number of output neurons is 1 - ---ί - ——- I hi hidden layer activation function double bending function ~ -1 output layer activation function ··--------------------- double bending Function '-'.S--1 Training sample number 628 pen ~ ~~ Number of test samples [ 314 pens --. Learning rate 0.6 --- Double twist function coefficient λ • ·—— - - -- -----_ _ _____ 1 ·------------------ Weight range [-0.5,05Ϊ~ ~~~~~ Training times 1000 times ^ _ 4.2 Sine function curve fitting experiment results Error

4.2.2 Function Approximation This experiment is a problem of fitting various different functions using the four-layer inverted transfer network of the hair of the county. The equation is (4.2)~(4 5). The scope of the training materials in this experiment is in the distribution of [Bu 6·28], and the distribution of the other words. The other parameters set in the case of the same value of the hardware and software are shown in Table 4.3. The experimental results are shown in Fig. 50~52, the blue sound ± surface formula... red for material 2, green for formula 3 and purple for formula 4. Table 4 d • Analysis table for post-experimental performance and precision for different systems 40 201232429. ^(jc) = i(sin(jc) + l) (4.2) y(x) = - (cos(j:) +1) 2 (4.3) 3»U) = i(l〇gW + 3) ( 4.4) j(jc) =-ί—exp〇) 600 (4.5) Table 4.4 Function of the approximation of the experimental results Analysis table network architecture 1-2-3-4 (hardware architecture) 1-2-3-4 (hard Body architecture) 1-2-3-4 (software architecture) Segmentation processing with or without hardware synthesis unit number 2 10 training times 62.298002 s 54.472789 5 435.489235 正 Sinusoidal function minimum error 1.2991e-5 7.0661e-5 3.2435 E-4 Sinusoidal function maximum error 0.0288 0.4950 0.2287 Sinusoidal function average error 0.0085 0.0742 0.0602 Cosine function minimum error 1.2991e-5 2.79l5e-5 1.6775e-4 Cosine function maximum error 0.0288 0.1170 0.1204 Cosine function average error 0.0085 0.0296 0.0521 Logarithmic function maximum Error 1.299le-5 1.2991e-5 3.9936e-6 Logarithmic function maximum error 0.0288 0.5149 0.5421 Logarithmic function mean error 0.0085 0.0238 0.0198 Exponential function minimum error l_2991e-5 2.0928e-5 1.8030e-6 Exponential function maximum error 0.0288 0.2238 0.2026 Index The average error of the function is 0.0085 0.0309 0.0288 ' --------

4.2.3 Battery residual power prediction This experiment uses the time delay neural network and the feedback neural network to predict the residual power of the battery discharge data obtained by the actual measurement. This information is for the battery at 40. C continues to record data at a current discharge of 2.4 amps. This will measure the voltage, current and battery temperature differently as shown in the formula (4 6) 41 201232429 ~ (4.8) as in the normalized way to let the input value between (7), i] as shown in Figure 54 The 50th is not the left side of the original data, the right side is the data processed by the formula. The parameters set by him should be as shown in Table 4.5 and Table 4.6. The experimental results are shown in the anus 17 to the figure ^10. The sum of the results of the seven and ninety-nine points and the five points of the Peng (4) results and the actual data, the blue in the figure is the predicted result curve, and the red is the actual discharge curve T. Table 4. 7 are two different networks Analysis table of efficiency and precision after road experiment. (4.6) (4.7) (4.8) :v-rl3 :(i + 2.5) -r 3 :r-r56 Name I When working β ί .... 1 Integer digits | ____— . - - ---------------- ~~ -» . - »>*_ , Decimal number Γ * ! One by one 铋 — _____ _·" 1 Input number of neurons one·--*1 i 丨delay number ί ___________- ί hidden layer neurons ί_________ number of output neurons [hidden layer activation function | rounded layer The function " ϋϋ number --- [ . Forecast sample number I learning rate i double bending tangent function system] number λ 权 weight value range training times 100MHz 16 bit ST — Ϊ 6 bits — time delay class neural network g a 3 ... — One three ones, five, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one, one ---- 0.6 [-0-5,0.57 ^ϊοοο^^~ Table 4.6 Battery residual power experiment feedback neural network related parameter setting 42 201232429

Table 4.7 Network Type Neural Network Network Architecture 54 Late Neural Network 9-5-1 (Hardware Architecture) Feedback Neural Network Segmentation Processing No 9-SJ (Hardware Architecture) 3-2-1 ( Hardware architecture) Number of feedback neural network hardware synthesis unit 10 training time each time 0.542483: 0-676606 no 10 - 0.429014 5 3-2-1 (hardware architecture) 2Ϊ 0.452101 minimum error maximum error mean error 2.7325e-5 0.0412 Μ0ΪΓ 2.7l48e-4 ~~οόοΓΓ 2.685 le-4 0.041 Factory 0.001 Factory 6.9390e-5 ~~〇0376 ~00004~ The above detailed description is for the specific description of the possible embodiments of the present invention, The actual example is not intended to cover the scope of the patent of the present invention, and the equivalent of her or the change of the technical spirit of the present invention shall be included in the patent scope of the present invention. Knowing that the case is not only innovative in terms of space type, but also can be added to the above-mentioned items, and can be submitted in accordance with the law, 恳喑 _ _ The main business must be patented * the board office approves the invention patent application, in order to invent the invention to the sense of virtue. Figure 1 is a CNAPS architecture diagram; Figure 2 is a schematic diagram of a two-dimensional pulsation array architecture; Figure 3 is a reverse-transfer-like neural network architecture diagram; Figure 4 is a symmetric hard-limit function diagram; Figure 6 is a double-bend tangent function diagram; Figure 7 is a linear function diagram; Figure 8 is a time-delay-like neural network diagram; Figure 9 is a feedback-like neural network architecture diagram; Figure 10 is a forward-operation unit model Figure 11 is a schematic diagram of the ring structure; Figure 12 is the forward operation architecture diagram; Figure 13 is the segmentation calculation flow chart; Figure 14 is the reverse operation architecture diagram; Figure 15 is the operation unit signal timing diagram; Figure 16 is a fixed-point decimal representation; Figure 17 is a hyperbolic function symmetry value diagram; Figure 18 is a double-bend tangent function symmetry value map; Figure 19 is a double-bend function symmetry value hardware architecture diagram; Twenty is a double-curved tangent function symmetry value hardware architecture diagram; Figure 21 is a fragment linear method hardware architecture diagram; Figure 22 is a fragment linear method diagram; Figure 23 is a fragment linear method modified architecture Fig. 24 is a schematic diagram of the piecewise linear method (1); 201232429. Fig. 25 is a schematic diagram of the piecewise linear method (2); Fig. 26 is a double bending tangent function structure diagram; Fig. 28 is the modified linear graph of the fragment; Fig. 29 is the error comparison graph of the fragment linear method; Fig. 30 is the PWL double bending tangent function graph; Fig. 31 is the PWL double bending Tangent function error graph; Figure 32 is the back-transfer neural network 1-2-3-4 architecture diagram; Figure 33 is the computational unit memory complement map; Figure 34 is the weight memory data configuration diagram ; _ stomach thirty-five is the weight _ deposit map (1); Figure 36 is the weight value storage map (2); Figure 37 is the operation unit collocation address diagram; Figure 38 is the target value diagram; Figure 39 Figure 490 is the control unit flow chart; Figure 40 is the hardware and Nios II architecture diagram; Figure 42 is the software planning flow chart; Figure 43 is the Ni〇s π embedded Processor hardware architecture diagram; • Figure 44 is the Avalon architecture diagram; Forty-five is the signal map for Avalon; Figure 46 is the Avalon read signal map; Figure forty-seven is the sine function curve fitting experiment through the neural network to output the training results. Figure 48 is the actual curve of the sine function Software and hardware fitting curve comparison chart; ·, Figure 49 is the sine function curve error analysis chart; = fifty is the function of the approximation experimental test data through the segmentation of the hardware network without the need for segmentation One is the approximation of the experimental hardware test data error analysis chart; ®52 is the function approximation of the experimental face through the tilt __ road map fifty-three for the approximation of the experimental software and hardware related heavy 峨 峨Fig. 45 201232429 Fig. Fig. 54 shows the original graph of the battery voltage and the input graph after the change; Fig. 55 shows the original graph of the battery current and the input graph after the change; Fig. 56 shows the original graph of the battery temperature and the input after the change. Figure 57 shows the comparison of the results of the battery discharge curve predicted by TDNN; Figure 58 shows the error results of the battery discharge curve predicted by TD 丽; Figure 59 shows the battery discharge curve predicted by the feedback neural network. FIG comparison result; FIG sixty is a feedback type neural network to predict the results of the battery discharge curve in FIG error; Main reference numerals DESCRIPTION

46

Claims (1)

201232429 VII. Patent application scope: L—a high-speed hardware reverse transfer and feedback type neural network with flexible architecture. The operation modes of the network include: three-layer reverse transfer-like neural network mode; four-layer inverted transfer class Neural network mode; standard feedback-like neural network mode; and feedback-weighted value-correctable feedback-like neural network mode. 1 2. As stated in the operating mode described in item 1 of the patent application, the content of the software update register can be used to determine the mode of operation of the network. 3. The hardware architecture of the high-speed hardware reverse transfer and feedback type neural network with elastic structure includes: Input device 'Miscellaneous device is a device that can generate digital jobs; can be customized hardware' Wherein the programmable hardware is connected to the input device and the data of the input device is processed and processed, the programmable hardware is an element programmable logic gate array; and the output device is The number of unrecognized interfaces is connected to (4) the data that can be programmed to calculate the 'device' is a device or microprocessor that can receive digital signals. 4. The operational mode described in the patent stipulations, wherein the standard feedback-like neural network mode comprises: a connection input layer; a feedback processing layer; 47 201232429 a hidden layer; Output layer; the feedback processing layer receives the hidden layer output of the previous-time iteration when input data, and then receives: 1 bucket, "α-the hidden layer of the person's iteration; the feedback layer sa between the hidden layers\ For the solid layer, the enrollment data of the hidden layer comes from the feedback processing layer and the input layer. The % % multiply accumulates the activation function of the hidden layer, and outputs the result to the output layer. After the multiplication and multiplication, the output of the output layer is obtained through the activation function of the output layer. # 申月Special ϋ The operation mode described in Item 1 is a feedback-reversible neural network mode. The architecture includes: a connection input layer; a feedback processing layer; a hidden layer; and a rounding layer; the feedback processing layer receives the hidden layer output of the previous iteration when inputting data, and then receiving The rounded feed material is passed to the hidden layer of the iteration; the weight value between the feedback processing layer and the hidden layer can be corrected by the inverse operation; the input data of the hidden layer comes from the feedback processing layer and the input input layer's weighted value. After the multiplication is added, the result is rounded out to the rounding layer through the activation function of the hidden layer; the output layer receives the multiplication of the output value of the hidden layer and the weight value, and then obtains the output of the output layer through the activation function of the output layer. The operation mode described in the first paragraph of the patent scope of June, wherein the three-layer reverse transfer type neural network mode includes: 48 201232429 an input layer; a hidden layer; and an output layer; The ring material image _, the butterfly (four) face transfer layer, · the input data of the hidden layer is multiplied by the weight value, after the hidden:, activation function, the result is output to the output layer; the input (10) is connected to the «Tibetan layer The round = with the weight of the county after the accumulation of the method, after the transmission of the _ _ record of the output of the round. 7 · As stated in the application of the 5th patent scope of the scope of the operation control + extended nuclear type of which the four layers Transfer-like neural network Road mode 'The mode structure includes: an input layer; a first hidden layer; a first hidden layer; and an output layer; the round-in layer receives an external t« when the input material is re-submitted (4) input: the trait is passed to the first a hidden layer; after multiplying and accumulating the weights of the input dragons of the hidden layer, the result is output to the second hidden layer through the activation function of the layer -1; the second hidden layer receives the hidden layer The output value is accumulated by the weight value county method and then passed through the activation function of the first hidden layer to transmit the result to the output layer; after the multiplication of the weight value of the output layer, the activation function of the round-trip layer is used. The rounding of the output layer. 8. As in the standard feedback-type neural network mode described in claim 4, the content of the control software update register can be used to determine the activation function used by the hidden layer and the output layer, respectively. . 49 201232429 9· 2 Please turn the patent to the _ weight value can be corrected in the feedback-type neural network horizontal type, and (4) the health update temporarily to determine the activation function used by the hidden layer and the output layer. . 10. If Shen Qingqing 5 Nina _ can repair the whistling type of neural network mode, the content of the control software update register can be used to determine the initial weight value used between the feedback processing layer and the hidden layer. U. The three-layer inverted transfer neural network mode described in item 6 of the patent application scope can be utilized. Control the contents of the software update register to determine the activation function used by the hidden layer and the output layer, respectively. 12. The four-layer inverted transfer neural network mode described in claim 7 of the patent application, the content of the control software update register can be used to determine the first hidden layer, the second hidden layer and the output layer respectively. Activation function. • A standard feedback type neural network mode as described in claim 8 of the patent application, wherein the activation function is a double bending function or a double bending tangent function. H. A feedback-type neural network callback mode in which the feedback weight value described in claim 9 is modified, wherein the activation function is a double bending function or a double bending tangent function. For example, the three-layer inverted transfer neural network mode described in item η of the patent scope includes the double bending function or the double bending tangent function. 16. The four-layer inverted transfer neural network mode of claim 12, wherein the activation function is a double bending function or a double bending tangent function. The activation function according to claim 13, wherein the double bending function comprises: a multiplier, the multiplier receives an input signal and multiplied by a fixed value of -1; 50 201232429 a first multiplexer Making a selection; the first multiplexer receives the rounding of the multiplier and the input signal and - adding the first adder value 2; the first adder receives the output of the first multiplexer and is fixed Receiving the first plus one nd-bit shift register, the output of the first two-bit shift register is shifted by two bits; a second adder that receives the round of the first adder and adds it to a first interpolated depth value; a second multiplexer 'the second multiplexer receives the second Red shift register output and a fixed value 〇 and make a choice; - three-bit shift register, the three-bit shift register receives the output of the second adder and makes three bits a third adder, the third adder receives the rotation of the second multiplexer and adds a second interpolation depth value and an output of the three-bit shift register; a third multiplexer that receives the output of the second multiplexer and the output of the three-bit shift register and makes a selection; - the second two-bit shift temporary storage The second two-bit shift register receives the output of the third adder and shifts the two bits; - the fourth multiplexer receives the third multiplexer The output and the output of the second bit shift register are selected as a selection; a subtractor that receives the output of the fourth multiplexer and subtracts from a fixed value 51 201232429 Dingkou, the fifth multiplexer receives the rounding of the subtractor and the output of the fourth multiplexer and makes a selection. The rounding of the fifth multiplexer is a double bending function. 18. The activation function as recited in claim 13 wherein the double f-curve function comprises: a multiplier 'the multiplier receives the input signal and multiplies the fixed value -i; The first multiplexer receives the output of the s-hai multiplier and the input signal and makes a selection; a first adder 'the first adder receives the output of the first multi-guard and adds the value to the - fixed value - the first - the interpolated depth value; the first multiplexer receives the The output of the first multiplexer and a fixed value of -1 are selected as a selection; the first bit shift register, the first bit shift register receives the output of the first force port and performs One-bit shift; a second adder, the second adder receives the output of the second multiplexer and adds the output of the first bit 兀 shift register and a second interpolation depth value; a multiplexer, the third multiplexer receives an output of the first bit shift register and an output of the second multiplexer; a second bit 7L shift register, the second one The bit shift register receives the output of the second adder and shifts by one bit; a fourth multiplexer receives the second bit shift register Output and the output of the second multiplexer and make a selection; 52 201232429 subtractor, the subtraction (four) receives the fourth multi-master round-out-fifth jade||, the fifth multiplexer receives the output and Making-selection; U-Yuan's turn-out and the killer The fifth multiplexer's turn is the double-bend tangent function. 19. The living tank according to claim 14, wherein the volume function comprises: - a multiplier, the multiplier receives an input signal and multiplies with a fixed value of 4; a first multi-guard: the first multiplexer receives the multiplier and the input signal and makes a selection; a first adder, the first adder receives the first multiplexer value 2 Adding a first shift to the first adder and taking the shift of the two bits; And rotating with a fixed value and a fixed-second adder, the second addition! I receives the round of the first adder and adds a value of a first interpolation depth; Two more 4's the second #1 receives the second-second shift register output and a fixed value of 0 and makes a selection; a two-bit shift register, the three-bit shift temporary storage Receiving the output of the second adder and shifting the three bits; a third adder, the third adder receiving the output of the second multiplexer and - the second interpolated depth value, the third The bit shift register is added by the wheel; the third multiplexer receives the wheel of the second multiplexer and the three bits 53 201232429 shift register Outputting and making a selection; - the second two-bit shift temporary storage n, the second two-bit shift register receiving the output of the third adder and shifting by two bits; - fourth a fourth multi-tuner receives the output of the third multi-guard and the output of the second two-bit shift register and makes a selection; - a subtractor, the thief receives the fourth The output of the multiplexer is subtracted from the - fixed value; - a fifth multiplexer that receives the wheel of the subtractor and the output of the fourth multiplexer and makes a selection; The output of the fifth multiplexer is a double bending function. 20. The activation function of claim 14, wherein the double bending tangent function comprises: a multiplier that receives an input signal and multiplies it by a fixed value _1. Working II, the first multiplexer receives the rounding of the multiplier and the input signal and makes a selection; - a first adder, the first adder receives the output of the first multiplexer and a fixed value -1, a first interpolation depth value is added; - a second multiplexer, the second multiplexer receives the output of the first multi-guard and a fixed value -1 and makes a selection: - - - bit miscellaneous, scale-bit shift (four) to touch the first addition | § output and make a one-bit shift; - second adder 'the second adder II receives the second more than 1 round And outputting the output of the first 54 201232429 one-bit shift register and a second interpolation depth value; the round-out third multiplexer receives the first _ ____ bit shift register and the output of the 5th multiplexer; the device receives the second plus-second-bit shift register, the second bit The output of the temporary storage device is shifted by one bit. - The fourth multiplexer receives the second-bit shift register and the third The wheel of the tool is turned out and made a choice; the output of the four multiplexer is combined with the - fixed value as a subtractor, and the s-hai subtractor receives the first subtraction; The output of the tool and the rounding of the subtraction are made as a choice; the turn of the fifth multiplexer of the magic 3 is a pair of bending tangent functions. 'If the application of the general section (4) mentioned in the article back to the roots ^ 垔 了 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回 回Corrected by calculation. 55