TWI417798B - High - speed reverse transfer neural network system with elastic structure and learning function - Google Patents

Description

High-speed inverted transmission neural network system with elastic structure and learning function

The invention relates to a high-speed inverted-transfer neural network system with elastic structure and learning function, in particular to a neural network system with highly parallel operation, in which nodes of each layer are connected to each other to form a capable A network that handles complex work.

In the past, traditional neural network computing systems were implemented on software. Many real-time control systems were limited by the speed of the software, and because of the speed limitation, the network system could not be huge in real-time applications, resulting in a system. The parameters can not get a lot of information, so that its performance can not be highlighted; in recent years, the concept of parallel processing has become more and more powerful, and the central processor has gradually evolved into parallel structure with the progress of the times and the bottleneck of technology, meaning that when parallel processing The time and performance of execution can be greatly improved. The current research indicates that many types of neural networks are gradually implemented by hardware, and the reason depends on the speed. Especially when the network system is large, the execution time is quite long if it is via software. Therefore, various kinds of neural network systems suitable for hardware have gradually expanded the research, and the research results seem to show a gradual increase, especially when the network training can replace the software, it is even more at the application level. A big breakthrough.

However, when network training can be implemented on hardware, hardware cost becomes an important indicator. If you use a lot of hardware, it seems that the economics and the software that takes time and length are unavoidable. In the ring-and-column architecture, it is only applicable to the multi-layer perceptron architecture. Although the control unit can be designed to control the number of layers, the network system is still limited by hardware. And many of the better learning algorithms also seem to point to the number of neurons with the number of iterations, which is dynamically adjusted to make it Road learning converges faster. For example, the common network pruning method, the vertical minimum flat method of the web-based base network, shows that the superiority of such learning algorithms cannot be demonstrated if the hardware architecture is fixed.

The patents on neural network technology are as follows:

1. U.S. Patent No. 5,087,826: The proposed architecture uses a multiplier input and a product of key values ( x.w ) for each neuron chain to form a two-dimensional array architecture. Although it is fast but consumes a lot of hardware costs and generates a large number of bus bars, it is not conducive to design.

2. CNAPS (Dan Hammerstrom, "A VLSI architecture for high-performance, low-cost, on-chip learning," Proceedings of International Joint Conference on Neural Networks, 1990, pp. 537-544. Dan Hammerstrom, Digital VLSI for Neural Networks, The Handbook of Brain Theory and Neural Networks, Second Edition, Michael Arbib, MIT Press, 2003.); The advantage of CNAPS is that each operation node has built-in addition and multipliers, and can be controlled by command bus input commands. Its operation, each node is equivalent to a simple arithmetic unit (Arithmetic Unit), so the algorithm for computing different types of neural networks is very flexible. Applying this architecture to the hardware development of neural networks, the number of nodes can be easily adjusted because the computing nodes use the same architecture. Unfortunately, the control instructions used in this architecture are very complicated, so it is necessary to compile the instructions with the software, and there is no hardware architecture dedicated to the activation function. Because of the universal architecture, the operation speed is sacrifice.

3. U.S. Patent No. 5,091,864: The proposed architecture does not have a bullet like CNAPS. Sexuality, but its architecture is simple and easier to design and use, which not only improves the speed of computing but also reduces the cost of development, but also simplifies the complexity of the control unit and shortens the development cycle. The input data is transmitted in a serial connection, that is, the data is first transmitted to the first operation unit, and the second operation unit is passed after two cycles, but because the data arrives at each operation unit at different times, It also adds to the difficulty of designing the control unit. The preferred part of this is the number of reduced activation functions. This architecture considers the characteristics of the one-dimensional array. The data is input and output with the pen pulse cycle. Therefore, the operation unit does not need to perform the calculation of the activation function at the same time, and then the activation function is taken out from the neuron and placed independently. In the backhaul part of the array, only one activation function needs to be designed to complete the operation, and it will not delay the operation. In addition, the architecture is designed with a set of shift registers, which can store the calculated data first and then transfer it back and forth. All the arithmetic units can immediately perform the next data calculation, which saves time. The disadvantage of this architecture is that there is no mechanism for learning parts, and only the retrieving operation can be performed for the network that is trained to complete.

4. U.S. Patent No. 5,799,134: The proposed architecture is similar to the concept of U.S. Patent No. 5,091,864, but the input data is connected in parallel, that is, all arithmetic units receive the same input signal at the same time. And by adding a subtractor and a multiplexer to the arithmetic unit, the change of the operation is more flexible. But there is also no mechanism for learning part, only the echo function of the neural network can be performed.

5. WO 2008/067676 A1: It proposes an adjustable artificial neural network system and architecture that performs forward and reverse transport of artificial neural networks in a segmented manner. The calculation and segmentation can be adjusted according to the needs. However, the hardware of the system must input the actual hardware chip, the number of segments, and the logic component usage limit into the software program, and then the hardware description program generates the required hardware description language code. Then download it to the hardware chip. Therefore, once the wafer is planned, the number of segments cannot be changed.

It can be seen that the above-mentioned conventional and current methods are not a good design and need to be improved.

In view of the shortcomings derived from the above-mentioned conventional methods, the inventor of the present invention has improved and innovated, and after years of painstaking research, finally succeeded in research and development of this high-speed inverted-transfer neural network system with elastic structure and learning function.

The object of the present invention is to provide a high-speed reverse transmission neural network system with elastic structure and learning function, which can provide a high-speed reverse transmission neural network system with elastic structure and learning function, and has the function of recalling and learning at the same time. Functional neural network system.

A high-speed inverse transfer neural network system with elastic structure and learning function for achieving the above object is composed of a forward operation block, a reverse operation block and a control unit; Architecture, the operation of the inverse-transfer-like neural network, which has the complete function of recalling and learning; the whole inverse-transmission network operation can be divided into forward operation and reverse operation, in the limited array of operation units, through The segmentation calculation method can train and recall the huge network system; since the neural network is very suitable for the calculation of hardware parallel processing, the present invention is single Instruction multiple data (Single instruction multiple data, SIMD) architecture design method, sharing the hardware array part signal, processing the inverse transfer network operation by command mode, and designing the ring string with high-speed pipeline design And segmented computing network system, designed to limit the hardware to synthesize a huge network system to replace the software that requires lengthy calculation; and the control unit calculates the reverse transmission network system by segmentation, fixing the hardware cost through The capacity of the memory determines the size of the network system, and the hardware design of the computing unit is based on pipelined and hardware multi-layer architecture, and the controller design is simplified by synchronous clock operation. In addition, the speed is above The improvement of the working clock is preferred, which makes the system as a whole more efficient. The present invention can improve the previous neural network hardware system, and achieve better performance while transmitting a small number of logical components and being flexible. Executive performance.

Please refer to FIG. 1 , which is a system diagram of a high-speed inverse transfer neural network system with an elastic structure and a learning function according to the present invention. As can be seen from the figure, the system mainly includes a forward operation block 11 and a reverse operation block 12 . The control unit 13 and the multiplexer are composed; when the forward operation is performed on the operation block 11, the data is weighted from the input layer via the hidden layer, processed by the activation function, and then transmitted to the output layer to calculate the network output value. , and each layer of neurons only affects the state of the next layer of neurons; if only the recall function is performed at this time, the result of the forward operation is switched through the multiplexer, and the calculation result is output; if the neural network learns, When the difference between the output result and the target value is too large, the forward operation result is converted into the input parameter of the reverse operation block 12, the reverse operation is performed, and the error signal is transmitted back. By modifying the weight value of each layer of neurons, the expected error is reached. Within the tolerance; in addition, the control unit 13 is used to control the flow of the overall system, that is, at the appropriate time The control signal is sent to control the operation sequence of the forward operation block 11 and the reverse operation block 12; since the operation of the reverse transfer type neural network is a multi-layer architecture, except for the input layer, the input of each layer is the previous layer. The output after the operation, plus each layer is the same operation mode, so the same operation unit can be used to perform different layer operations, that is, the operation of the first node of the hidden layer using the arithmetic unit is also used. The same arithmetic unit performs the operation of the first node of the output layer. In this way, only a number of arithmetic units equal to the maximum number of single-layer hidden layers are needed, and multi-layer reverse transfer-like neural network operations can be performed, which is greatly reduced. The amount of hardware used; since the operation unit array completes a single layer, the operation result will continue to be used for the operation of the next layer. Therefore, the present invention directly transfers the operation result to the input data bus to perform the operation of the next layer to form a ring. Architecture, so the data can be directly input for calculation, and no need to be processed through the control unit, which will help shorten the operation time. between.

Please refer to FIG. 2A and FIG. 2B, which are schematic diagrams of segmentation calculation and hardware co-construction of a high-speed inverse transmission neural network system with elastic structure and learning function according to the present invention, which can be divided in the whole inverse network operation. For the forward operation and the reverse operation, in the SIMD architecture design mode, the hardware array part signals are shared, and the operation of the inverse transfer network is processed in an instruction manner; the invention is designed and improved, and the high-speed pipeline design method and the segmentation calculation are performed. The network system is designed with more complete hardware to replace the software that requires lengthy calculation. The improved segmented computing architecture is shown in Figure 2A. The segmentation calculation method can still make the hardware architecture still have a ring. The serial string feature only needs to perform segmentation calculation and weight position design when controlling the reverse network process; for a single-layer network, for example, if there are 7 hidden layers, and the operation unit in the forward operation block If the number of serials is three, the calculation must be performed in 3 steps to complete the calculation. In addition, since the reverse transmission network is divided into forward operation and reverse operation during the whole training process, and the forward operation block is calculated, The direction is idle, and vice versa; wherein the forward operation is dominated by the array of arithmetic units, while the arithmetic unit is multiplied and added to calculate the weighted sum of the neurons; and in the reverse calculation process, the calculation is hidden. Layered At that time, it can be found that the multiplication and addition operations are used in the calculation process, only the difference in the weight order and the input data, and the activation function is not required. Therefore, the operation unit in the forward operation can be provided to the reverse process through the configuration memory. Calculate the use of hidden layer error terms; based on the order relationship, the memory used is in the queue mode; and the input signal for calculating the hidden layer error term needs to be read from the δ伫 column to the input bus of the arithmetic unit array. However, because the system adopts flexible architecture design, the array of arithmetic units may be smaller than the number of nodes in the actual network. In this case, segmentation calculation is required; whether the array of computing units is larger than the network system, it must be stored in the memory first. Since the output data of the neurons stored in the hidden layer of the hidden layer has been read before the reverse operation, the hidden layer array can be used to store the result of the operation unit array in the reverse process, which does not delay the operation process, and can Saving the usage of logic components, as shown in Figure 2B, the forward operation and the reverse operation signal go to the same; Computing an outer segment of the present invention may be changed dynamically set according to the needs of the hardware in the synthesis.

Please refer to FIG. 3A and FIG. 3B, which are diagrams of the operation unit system and the operation unit signal timing diagram of the high-speed inverse transmission neural network system with elastic structure and learning function, and the internal structure of the operation unit can be mainly divided into The three parts are respectively memory (the memory of the stacked architecture 1411, the memory 1412 of the array structure), the multiply accumulator 142, and the shift register 143, as shown in the figure, all three parts are Independent operation, and pipeline design, the operation is dismantled independently, which can greatly improve the performance. In the forward operation block, many arithmetic units are connected in series, and the operation unit is equivalent to Neurons mainly multiply the weight value and the input value by multiplication; in the forward operation, the weight of the hidden layer is used first, followed by the output layer, but in the reverse operation, the δ priority of the output layer is calculated. The weight correction amount of the opposite output layer and the modified weight are also calculated preferentially from the hidden layer of the output layer. Since the memory mode in which the input layer to the output layer is heavily weighted is a queue, after the weight value is read, In order to match the forward operation in the learning process, the corresponding weights can be read out in the correct order, and in this operation unit, a memory 1411 having a stack structure is internally designed to store the weight values in order. It is not necessary to control the memory content through an additional address line; in the reverse process, since the error value required for calculating the δ of the output layer is also a multiply-and-accumulate operation, the calculation is performed using the hardware co-structural characteristic. The weight of the output layer connected to the hidden layer is stored inside the operation unit, and the weight value order is completely different and the weight of the hidden layer partial weight is not required. The layer is used as the input value, and the hidden layer is used as the output of the output layer neuron. Therefore, in addition to the stacking mode memory used by the forward operation, the operation unit needs to design another matrix memory to calculate the inverse operation. The required weight value; because two types of memory modes are arranged inside the arithmetic unit, an additional multiplexer 144 is required for selection, and in order to simplify the design of the controller, there is a The additional addressing mode is used to write the memory 1411 of the stacking structure of the weight bus 15 to the memory structure of the stacking structure of the computing unit, or the memory 1411 or the queue structure of the stacked architecture in the computing unit. The write signal of the memory 1412 must be addressed with the operation unit, and the memory 1411 of the stacked architecture or the memory 1412 of the queue structure can be shared because the processing data is processed in parallel; because the forward operation block String of arithmetic units After the calculation, the calculation and multiplication operations are performed simultaneously. When the calculation is completed, the calculated values in the respective operation units are transferred to the next level through the shifting of the meta-register. Therefore, many registers are used internally in the arithmetic unit to solve the problem of asynchronous signal processing. The hardware synchronization signal can simplify the design of the controller.

Please refer to FIG. 4A and FIG. 4B, which are system diagrams and operation timing diagrams of the forward operation block of the high-speed inverse transfer neural network system with elastic structure and learning function according to the present invention, and the structure is a multi-layer perceptron architecture. Therefore, each layer of operation has a relationship between the front and the back. If the hardware is also designed in layers, the cost and logic components will be proportional to the network system, so the front layer of the network can be connected from the input data. 16 given data, and the result of the back-level operation is stored in memory. In the next layer of operation, the stored value is placed on the input data bus 16, which can greatly reduce the amount of hardware used; The operation block is actually composed of a plurality of operation units 14 in combination. The more the operation unit 14, the larger the number of forward networks can be calculated at one time, but the number of the synthesis units 14 is often constrained by the actual hardware. If you want to have a flexible system, you need to store the result of the operation. This design can not only synthesize a huge network system, but also the flexible design is suitable for a variety of nerves. The road algorithm; as shown in Fig. 4A, there is a multiplexer 18 for controlling whether the operation result has to be converted by the activation function 17, and the purpose is to calculate the error term (Error) due to the hardware co-construction in the reverse operation. There is no need to convert through the activation function 17, but the calculation of the hidden layer error term is also limited by the number of arithmetic unit synthesis, so the value needs to be stored; in addition, the forward operation block is used every time without considering the output delay time. The cycle of pulse completion is shown in Figure 4B. When the multiply accumulator is completed, the time to wait is the number of input nodes. Quantity, so a clock can calculate an input node multiplied by the internal memory weight value of the arithmetic unit array, and when the signal transmission of all input nodes is completed, the Shift signal must be enabled immediately to A node calculation value is transferred to the activation function 17 point by point, and the transmission waiting clock is proportional to the number of operation units; in addition, in the forward operation block, the input value of the operation unit array and the activation function 17 are passed. The subsequent output values are respectively transmitted to the two levels of the Layer Input Bus 19 and the Layer Output Bus 20, in order to make the hierarchical input stack and the hierarchical output of the reverse operation block. The values stored in the two bus bars are stacked to calculate the neuron input and output required for training.

Please refer to FIG. 5A and FIG. 5B, which are a structural diagram and a stacking timing diagram of a reverse operation block of a high-speed inverse transfer neural network system with an elastic structure and a learning function according to the present invention. The reverse operation is divided into four categories. For calculating δ, Error , △ w and weight update respectively, when calculating the error term of the hidden layer, the arithmetic unit array in the forward operation block is used, and only the error term of the hidden layer is calculated and transmitted back to the forward direction due to the reverse operation. Operation, therefore, using the data flow (Data Flow) and pipeline mode, the reverse operation block is queued or stacked to read signals or writes step by step; in the forward operation block, the input values of the neural network layers are And the output value is stored, wherein the input value is segmented, so the serial input bus 26 will import data due to the segmentation calculation, so in order to simplify the memory capacity of the input level stack, design one The flag signal is used to control the writing of the input level stack. The flag is only started when the segment calculation is performed. As shown in Figure 5A and Figure 5B, the output value of the neural network is saved. Into the hierarchical output stack, where the hidden layer queue only stores the value of the hidden layer neuron output; after the current direction is completed, the δ is first calculated first, and the calculation δ can be divided into two parts. versus , where j is the number of output nodes, and i is the number of hidden layer nodes. The control unit sends the hierarchical output stack in the learning block to read the signal. After 2 clock cycles, the ideal output value D _{j of the} training sample is sent and the data is also outputted from the output stack in sequence; The time is sent to the subtractor 21 and the multiplier 22 to calculate and And then sent to the multiplier 22 to find The ideal value of the training sample is also obtained. Finally, the two operation results are sent to the multiplier 22 for operation, and the output layer is obtained. The Error Select signal is used to select different calculations. When the value is n is the output layer, Error Select=0, otherwise a 1 signal is given; when the output layer is After the calculation is completed, the data will be stored in the δ伫 column and transmitted to the input data bus of the PE array for calculation. However, it is necessary to pay attention to whether the number of physical operation units is enough to hide the number of layers. Therefore, the δ伫 column needs to have a re-read function for transmission to the input data bus of the PE array, and the weight of the PE queue. The value is already configured at the beginning of the weight initialization. The result of this PE array operation does not need to be passed to the activation function for calculation, so the transfer enable (Transfer Enable) is set to 0; , is stored in the hidden layer first-in first-out queue (Hidden FIFO); and then the layer output is stacked to read the signal to maintain the number of hidden layer nodes, read out the output of the hidden layer, Calculation, and It is read by the Hidden FIFO, and then the data is transferred to the multiplier 22 for operation to obtain a hidden layer. The hidden layer δ is similar to the output layer δ algorithm. The layer output stack must first read out the signal. The signal read at this time is the hidden layer neuron output signal. The other difference is the hidden layer δ when calculating the error term. It is necessary to use the array of operation units of the forward operation block and the δ伫 column also needs to have the function of repeated reading. When the δ of the output layer and the hidden layer are calculated, and all are stored in the δ伫 column, the process starts. Calculate △ w ; Since the number of Δ w is (the number of input layers + 1) × the number of hidden layers + (the number of hidden layers + 1) × the number of output layers, the pipeline design is used to increase the speed; Simultaneously reading, the input multiplier 22 is multiplied by the learning rate η, and then sent to the multiplier 22 multiplied by the hierarchical input stack having repeatable reading, and finally multiplied by the inertia factors η _m and Δ w The subsequent results are added together, and while the new △ w is calculated, the Δ w伫 column will also store the value; in addition, since the system uses batch training, the adder 23 and the batch queue can replace all The weighted accumulator of the number of weights, not reached the next time Era, △ w accumulation storing the accumulated content to the batch queue, and before each batch of training, the control unit writes a randomly generated via linear feedback shift register (Linear Feedback Shift Register, LFSR) The disturbance amount [-2 ^-10 , +2 ^-10 ] is in the batch queue, so that the algorithm can jump out of the local solution, and the place where the error is calculated is to place the accumulator 24, after each batch training The comparator 25 is used to check if the current error is within the allowable range.

Please refer to FIG. 6 , which is a flow chart of a control unit of a high-speed inverse transmission neural network system with elastic structure and learning function according to the present invention. The control unit is responsible for controlling the entire system operation, and all the numerical calculations are stacked or queued. The content is stored, so the controller only needs to write and read each queue and stack in a finite state machine to control the timing and current state, and in addition to the storage component, the controller needs to calculate the segmentation operation. Accumulator clearing, multiplexer signal selection line and arithmetic unit array position; and input sample queue, output sample queue and result storage queue, controller is used as start signal; as can be seen from the figure, control unit The process is divided into three blocks, in which the initialization generates the start signal. When the stack is initialized, the system immediately performs the forward operation, which can shorten the waiting time. In addition, the initialization update of the queue can utilize the forward operation process. It can be operated; if the initialization of the queue is not completed in the reverse operation, it will wait for the queue to be initialized before the reverse operation. When performing the forward operation, it first evaluates whether the segmentation calculation is needed, and then passes the activation function, and then judges whether it is the output layer. If not, it means that the hidden layer is currently calculated to the output layer, so it will also Evaluate whether the segmentation calculation is performed; after the activation function is sent, it will judge whether it is the recall state. If not, it will wait for the operation unit to complete the output signal δ, and then calculate the output layer δ. At the same time, the hidden layer δ is successively calculated, but due to the hardware co-construction relationship, it is also necessary to consider whether to use the segmentation calculation to calculate the error term, but only need to calculate Δ w and update the weight to complete a training.

Please refer to FIG. 7A to FIG. 7D, which are timing diagrams and state diagrams of weight storage of the high-speed reverse transmission neural network system with elastic structure and learning function according to the present invention, and the weights are initialized due to the use of fixed hardware. Number, so the weight value must be used as a match to match the calculation input, so that the operation can be used normally, and the weight initialization is not only used for initialization. It is calculated according to the designed weight for each iteration. The configuration mode is stored in the queue or stacked, so that the calculation can be made without error; at the beginning of the system, the initialization action must be performed first, and the randomly generated [-0.5, +0.5] weights are stored in the stack in each operation unit. And queues for recall and learning use; as shown in Figure 7A, if the number of physical units is 3, the network system to be synthesized is a 2-7-5 layer 2 network, the control unit It will be initialized according to the network system parameters given by Nios II; in order to match the order of signal reading, it will be written in reverse order from the weight value of the last node of the output layer. ~ When writing to the PE ₂ stack, the number of output layers is divided by the number of synthesis units, and the position of the PE array is successively decremented. Similarly, the weight value of the next output node ~ Stored in the PE ₁ stack. After being written to the PE ₁ stack, the next write location is PE ₃ ; when the hidden layer writes to the output layer, the receiver begins to write the weight of the input layer to the hidden layer. At the same time, the PE array also records the current position in each stack, which is convenient for calculating the second layer network usage in the forward operation. As shown in Figure 7B, the initial PE stack position written is the number of hidden layers. The arithmetic unit synthesizes the remainder, that is, PE ₁ starts writing the last node weight value of the hidden layer. ~ Write in reverse order, and when writing to the PE ₁ stacking position, decrement from the highest position until the last node is written. Note that when writing the first value, the arithmetic unit The stack in the record records the read address in each array. When processing the segmentation calculation, the order of the weight values can be correct. In the learning process, the weight value is also used for calculation, so the weight value is used. When writing to the PE array, it must also be written into the weight FIFO in the learning unit block, that is, (2+1)*7+(7+1)*×5 total 61 The weight value is dispersed in the stack of the PE ring array, and the learning block is placed in the weight value sequence. In addition, in the reverse operation process, the PE array also needs the weight value calculation because of the hardware co-construction relationship. However, the order of storage and reading is completely different from that in the forward process, so the weight value of the learning block is repeatedly read out, and in the reverse process, the PE array operation does not require bias (Bias), so when Bias After the weight value is read from the weight value column, the PE column is written. The signal is set to 0, and its writing mode is as shown in Figure 7C. When the segmentation calculation network is more than once, the weight value is read more times. When the weight value is initialized, the operation unit is internal. The weight storage is shown in Figure 7D.

Please refer to FIG. 8A to FIG. 8D, which are schematic diagrams and timing diagrams of the forward operation segmentation of the high-speed inverse transfer neural network system with elastic structure and learning function according to the present invention. After the weight value stack is initialized, the following is performed. It is the forward operation of the inverse neural network. The forward operation can be divided into the number of layers and the segmentation. Each layer of network operation needs to consider whether to use segmentation calculation. When the control unit transmits the stack read in the PE array. The access signal reads the weight value, and the Bias is stored on the input data bus, and then the input sample is read out. Since the input queue signal contains the input value of Bias (2+1)=3 inputs, it is required. The three clock cycles are maintained, but since the number of hardware units in the arithmetic unit is only three, the first three nodes can only be calculated first, as shown in Step 1 in Figure 8A. When the operation is completed, the operation result is stored. Go to the hidden layer queue and the hierarchical output queue, and then perform the segmentation operations of Step2 and Step3. Note that Step3 is the last segmentation calculation, so the Bias and input sample queues in the input layer will be stored first. Stage input queue, used for the reverse operation, and is a timing chart of FIG Step1 eight of B, C, and additionally the previous figure eight eight D compared to FIG operation timing chart.

Please refer to FIG. 9A to FIG. 9C, which are schematic diagrams and timing diagrams of the inverse operation segmentation calculation of the high-speed inverse transmission neural network system with elastic structure and learning function according to the present invention. Before performing the reverse operation, it is necessary to judge the initialization. Whether the PE queue has been written, because the forward operation starts after the PE stack is written, but the PE queue needs to repeatedly read the weight value queue in the learning block, so the time spent is If it is judged that the number of segments is large, it is likely to be greater than the forward operation. Before calculating the correction amount of the weight value, it must be calculated In the forward operation process, the control unit has separately stored the input and output of each layer into the hierarchical input and hierarchical output stack in the learning unit; wherein the calculation δ can be divided into two parts. versus , where j is the number of output nodes, and i is the number of hidden layer nodes. The control unit sends the Layer Output Stack read signal in the learning block. After 2 clock cycles, the ideal output value D _{j of the} training sample is sent and the data is also output from the output stack. At this time, it is sent to the subtractor and the multiplier for calculation. and And then sent to the multiplier to find . The ideal value of sending the training sample at the same time has also been obtained. Finally, the two operation results are sent to the multiplier operation, and the output layer is obtained. And as shown in FIG. 9A, wherein the Error Select signal is used for calculation, and when the value is n is the output layer, Error Select=0, otherwise a 1 signal is given; and when the output layer is After the calculation is completed, the data will be stored in the δ伫 column and transmitted to the Input Data Bus of the PE array for calculation. However, it should be noted at this time whether the number of physical operation units is sufficient for the number of hidden layers as shown in FIG. 9B. Therefore, the δ伫 column needs to have a re-read function for transmission to the Input Data Bus of the PE array, and the weight of the PE queue. The value is configured at the beginning of the weight initialization. The result of the PE array operation does not need to be passed to the activation function for calculation, so Transfer Enable is set to 0; , in turn, stored in the Hidden FIFO, then the Layer Output Stack read signal maintains the number of hidden layer nodes, reads the output of the hidden layer, and performs Calculation, and It is read by the Hidden FIFO, and then the data is transferred to the multiplier operation to obtain the hidden layer. In addition, calculate the weight correction amount Must be calculated first versus Initially, the δ伫 column is first reset, because the PE array operation has been read in the reverse process. As can be seen from Figure 9C, the control unit will first send the Layer inputrd=1 signal to read the input value of the output layer. At this time, multiply the learning rate η After one clock cycle, set delta w FIFO rd=1, and take it out from the △ w伫 column Multiplied by the inertia factor η _m , and then the control unit reads a sum from the δ 伫 column And calculate all the input values of the layer .

Please refer to FIG. 10 , which is a flow chart of the reciprocating clock of the high-speed inverse transmission neural network system with elastic structure and learning function according to the present invention. It can be seen from the figure that the arithmetic unit synthesis number and the neural network system generate four different combinations. After the activation function operation, if the arithmetic unit is synthesizing enough, it only needs to wait for the neuron output value to pass through the ring structure, and the waiting time is the neuron output node of the layer; if the synthesis number is insufficient, it will wait The time is the completion of the segmentation operation plus the remainder of the layer divided by the arithmetic unit; if the remainder of the layer is 0, the remainder of the layer is the composite number of the arithmetic unit, and the quotient of the layer is decremented by 1; Table 1 shows the number of clocks required for each process. It can be used to find equations (1) to (4), where the unit is clock cycles, each clock is 10 ns, and I is the number of input layer nodes. H is the number of hidden layer nodes, O is the number of output layer nodes, MaxPE is the number of arithmetic unit synthesis, H _MOD is the quotient of the hidden layer division unit, and H _REM is the remainder of the hidden layer division unit synthesis number, O _MOD Output layer In addition to the quotient of the number of synthesized units, O _REM is the remainder of the output layer divided by the number of units synthesized; the use of equations (1) to (4) is: where the number of synthesized units is sufficient for the hidden layer and the output layer, Use the formula (1) 41 + I + 2. H + O , and when the unit synthesis number is sufficient for the hidden layer but insufficient for the output layer, use the formula (2) 41 + I + 2. H + O _REM + O _MOD . (8+ H +2. MaxPE ), in addition, when the unit synthesis number is sufficient for the output layer but not enough for the hidden layer, use the formula (3) 41 + I + H + O + H _REM + H _MOD . (8+ I +2. MaxPE ), if the unit synthesis number is not enough for the hidden layer and the output layer, use the formula (4)41+ I + H + H _REM + O _REM + H _MOD . (8+ I +2. MaxPE )+ O _MOD . (8+ H +2. MaxPE )

The high-speed reverse-transfer neural network system with elastic structure and learning function provided by the invention has the following advantages when compared with other conventional technologies:

1. Since the present invention is based on the use of a controller with a reverse transfer hardware to develop a perfect function of the reverse transfer network, and the use of limited hardware costs, the size of the discarded network system is limited by the concept of the arithmetic unit array, the use In the segmentation calculation mode, a huge reverse transfer network operation can be performed.

2. The hardware implementation of the present invention uses a pipeline with a multi-parallel computing architecture. By using these methods, the overall computing speed is improved, and the user can easily evaluate whether it is sufficient to apply on the instant control system according to the user's needs. Customized.

3. Since the error degree and the accuracy difference are quite small, and the operation speed is much faster than that under the software, the inverse transfer network hardware design of the present invention is sufficient to replace the software calculation, and can be applied to the lower order at the application level. Embedded systems are implemented in a traditional accelerated interface card.

4. The most important feature of the present invention is a highly parallel computing architecture, and the nodes of each layer are connected to each other to form a network capable of handling complex work; the present invention has the function of recalling and online learning, and improves the neural network of the past. The hardware system, while the clock is 100Mhz and the control unit can segment the back-transfer network without limiting the hardware cost, without re-planning and designing the entire system.

The above description is intended to be illustrative of a possible embodiment of the invention, and is not intended to limit the scope of the invention. In the scope of patents.

To sum up, this case is not only innovative in terms of technical thinking, but also able to enhance the above-mentioned multiple functions compared with conventional articles. It should be submitted in accordance with the law in accordance with the statutory invention patents that fully meet the novelty and progressiveness, and you are requested to approve this article. Invented the patent application, in order to invent the invention, to Chengde.

11‧‧‧ Forward Operation Block

12‧‧‧Reverse operation block

13‧‧‧Control unit

14‧‧‧ arithmetic unit

1411‧‧‧Stacked memory

1412‧‧‧ Memory of the array architecture

142‧‧‧Multiply accumulator

143‧‧‧Shift register

144‧‧‧Multiplexer

15‧‧‧weight bus

16‧‧‧Input data bus

17‧‧‧Activation function

18‧‧‧Multiplexer

19‧‧‧Level input data bus

20‧‧‧Level output data bus

21‧‧‧Subtractor

22‧‧‧Multiplier

23‧‧‧Adder

24‧‧‧ accumulator

25‧‧‧ Comparator

26‧‧‧Serial input bus

FIG. 1 is a system diagram of an implementation of a high-speed inverse transfer neural network system with elastic structure and learning function; FIG. 2A is a sectional calculation of a high-speed inverse transfer neural network system with elastic structure and learning function according to the present invention; FIG. 2B is a schematic diagram showing the hardware co-coordination of the high-speed inverse transmission neural network system with elastic structure and learning function; FIG. 3A is a high-speed inverse transmission neural network with elastic structure and learning function according to the present invention; The operation unit system diagram of the system; FIG. 3B is a timing diagram of the operation unit signal of the high-speed inverse transmission type neural network system with elastic structure and learning function; FIG. 4A is a high-speed analysis of the elastic structure and the learning function of the present invention. A system diagram of a forward-looking computing block before transmitting a neural network system; FIG. 4B is a timing diagram of a forward-running operational block of a high-speed inverse-transfer neural network system with elastic structure and learning function; FIG. The invention discloses a structural diagram of a reverse operation block of a high-speed inverse transfer type neural network system with elastic structure and learning function; FIG. 5B is a flexible structure and a learning function of the present invention. The stacking timing diagram of the fast reverse transfer type neural network system; FIG. 6 is a flow chart of the control unit of the high speed reverse transfer type neural network system with elastic structure and learning function; FIG. 7A is the elastic structure and learning of the present invention The write timing diagram of the hidden layer to the output layer stack of the functional high-speed inverse transfer type neural network system; FIG. 7B is the input layer to the hidden layer of the high-speed inverse transfer type neural network system with elastic structure and learning function of the present invention The write timing diagram of the stack; FIG. 7C is a timing sequence diagram of the write sequence in the operation unit of the high-speed inverse transfer neural network system with elastic structure and learning function; FIG. 7D is an elastic structure and learning of the present invention The weighted value initialization state diagram of the functional high-speed inverse transmission-like neural network system; FIG. 8A is a schematic diagram of the segmentation calculation of the forward operation of the high-speed inverse transmission-like neural network system with elastic structure and learning function; FIG. The signal relationship diagram of the segmentation calculation of the forward operation of the high-speed inverse transmission type neural network system with elastic structure and learning function of the present invention; FIG. 8C is a bullet of the present invention The first layer operation timing diagram of the high-speed inverse transfer type neural network system of structure and learning function; FIG. 8D is the first step of the forward operation of the high-speed inverse transfer type neural network system with elastic structure and learning function The second layer operation timing chart; FIG. 9A is a δ signal diagram of the high-speed inverse transmission type neural network system with elastic structure and learning function; FIG. 9B is a high-speed reverse transmission type nerve with elastic structure and learning function according to the present invention; Schematic diagram of the segmentation calculation of the reverse operation of the network system; FIG. 9C is a Δw signal diagram of the high-speed inverse transmission neural network system with elastic structure and learning function; and FIG. 10 is an elastic structure and learning of the present invention A functional high-speed reverse-transfer-like neural network system recalls the clock flow chart.

11‧‧‧ Forward Operation Block

12‧‧‧Reverse operation block

13‧‧‧Control unit

Claims (15)

A high-speed inverse transfer neural network system with elastic structure and learning function, the system mainly comprises: a forward operation block, when performing forward operation, data is processed from the input layer via a hidden layer weighting operation through an activation function After that, the network output value is calculated to the output layer, and each layer of neurons only affects the state of the next layer of neurons. If only the recall function is performed at this time, the result of the forward operation is switched through the multiplexer. Outputting the calculation result; a reverse operation block, if the neural network learns, and the output result is too different from the target value, the forward operation result is converted into the input parameter of the reverse operation block, and the reverse operation is performed, and The error signal is transmitted back, and the expected error is within the tolerance range by modifying the weight value of each layer of neurons; a control unit is used to control the flow of the overall system and transmit control signals to control the forward operation block and reverse The operation sequence of the operation block; the operation of the high-speed inverse transfer type neural network system with elastic structure and learning function is a multi-layer architecture, except Outside the layer, the input of each layer is the output of the previous layer operation, and each layer is the same operation mode, so the same operation unit can be used to perform different layer operations; After the single layer is completed, the operation result will continue to be used for the operation of the next layer. Therefore, the high-speed inverse transfer type neural network system with elastic structure and learning function directly transfers the operation result to the input data bus to perform the next layer operation. Forming a ring structure, the data can be directly input for calculation, and no further processing is required via the control unit. High-speed inverted transfer type with elastic structure and learning function as described in item 1 of the patent application scope The neural network system, wherein the inverse transfer type neural network operation can be divided into forward operation and reverse operation, and the single-instruction multi-data bus (SIMD) architecture design mode is used to share the hardware array part signals to instruct the instruction The way to handle the operation of the inverted network. The high-speed inverse transfer neural network system with elastic structure and learning function as described in claim 1 wherein the arithmetic unit includes at least a memory, a multiply accumulator, and a shift register. All three parts are operated independently, and the pipeline design method is adopted, and the operation and disassembly are independently performed to greatly improve the performance. For example, the high-speed reverse-transfer neural network system with elastic structure and learning function described in claim 1 of the patent scope, wherein two types of memory modes are arranged in the internal unit of the operation unit, so an additional multiplexer is required. To select, and in order to simplify the design of the controller, there is an additional addressing method in each arithmetic unit for writing the weight bus to the memory of the stacked architecture or the memory of the queue architecture in each computing unit. . The high-speed inverted-transfer neural network system with elastic structure and learning function as described in claim 1, wherein the computing unit is designed with a stacked architecture memory and a matrix architecture memory; and the stacked architecture The memory system is used to store the weight values in order, without the need to control the memory contents through additional address lines. In addition, the array structure memory is used to calculate the weight values required for the reverse operation. The high-speed inverse transfer neural network system with elastic structure and learning function as described in claim 1 of the patent scope, wherein the forward operation block is serially connected by the operation unit, so each operation unit calculates the multiplication and addition. The calculation is performed simultaneously. When the calculation is completed, the calculated values in the respective operation units are transferred to the next by means of the shifting element register. Level. For example, the high-speed inverse transfer neural network system with flexible structure and learning function as described in claim 1 of the patent scope, wherein the control unit is responsible for controlling the entire system operation, and is mainly responsible for storing components, calculating segmentation operations, and computing unit array positions. And use as a start signal. The high-speed inverted-transfer neural network system with flexible structure and learning function as described in claim 7 of the patent application, wherein the storage component means that the control unit needs to write or read each queue and stack respectively. A finite state machine controls the timing and current state. The high-speed inverse transfer neural network system with flexible structure and learning function as described in claim 1 of the patent scope, wherein the calculation of the segmentation operation means that the control unit needs to be responsible for the accumulator clearing, the multiplexer signal selection line and the operation. Unit array position, etc. For example, the high-speed inverse transmission neural network system with elastic structure and learning function as described in claim 1 of the patent application, wherein the use as the initial signal means that the input sample is queued, the output sample is queued, and the result is stored. Column, the control unit is used when the start signal. For example, the high-speed inverse transmission neural network system with flexible structure and learning function described in claim 1 of the patent scope, wherein the flow of the control unit can be roughly divided into: (1) generating a start signal, when the stack is initialized The system immediately performs the forward operation, which can shorten the waiting time, and the initialization update of the queue can be operated by the forward operation; (2) if the initialization of the queue is not completed when the operation is reversed, It will wait for the reverse operation after the initialization of the queue; (3) When performing the forward operation, it will first evaluate whether the segmentation calculation is needed, and then pass the activation function, and then judge whether it is the output layer, if not, it represents At present, the hidden layer to the output layer is calculated, so the segmentation calculation is also evaluated first; (4) after the activation function is sent, it is judged whether it is currently the recall state, and if not, it will wait for the arithmetic unit to wait for the signal to be completed. Then, the output layer δ is calculated, and when the δ of the output layer is calculated, the hidden layer δ is successively calculated, but due to the hardware co-construction relationship, it is also considered whether segmentation is used here. Calculate to calculate the error term, but only need to calculate △ w and update the weight to complete a training. The high-speed inverse transmission neural network system with flexible structure and learning function as described in claim 11 of the patent scope, wherein the forward operation system is a multi-layer perceptron system, so each layer operation has a relationship of front and back layers, The front layer of the network can be given the given data from the input data, and the result of the later layer operation needs to be stored by the memory. In the next layer of operation, the stored value is placed on the input data bus. The high-speed inverse transfer neural network system with elastic structure and learning function as described in claim 11 of the patent scope, wherein the reverse operation is divided into four categories, namely, calculating δ, Error , Δ w, and weight update, respectively. When calculating the error term of the hidden layer, the array of the operation unit in the forward operation block is used, and the data stream and the pipeline mode are used to gradually read or write the signal or the stack of the reverse operation block. For example, the high-speed inverse transfer neural network system with flexible structure and learning function described in claim 11 of the patent scope, wherein after the activation function is sent, the arithmetic unit synthesis number and the neural network system generate four different combinations. The four combinations are respectively the arithmetic unit synthesis number is sufficient for the hidden layer and the output layer, and the arithmetic unit synthesis number is sufficient for the hidden layer but insufficient for the output layer, The unit synthesis number is sufficient for the output layer but the hidden layer and the arithmetic unit synthesis number are not enough for the hidden layer and the output layer. If the operation unit synthesis is sufficient for the hidden layer and the output layer, only the neuron output value needs to wait for the ring structure. The waiting time is the neuron output node of the layer; if the composite number is less than either or both of the hidden layer and the output layer, the waiting time is the segmentation operation plus the layer division operation The remainder of the unit's composite number. For example, the high-speed inverse transfer neural network system with flexible structure and learning function as described in claim 14 of the patent application, wherein when the arithmetic unit synthesis number is sufficient for the hidden layer and the output layer, waiting for the neuron output value to pass through the ring structure Time, use the following formula to calculate the waiting time: 41 + I + 2. H + O , and when the unit synthesis number is sufficient for the hidden layer but insufficient for the output layer, wait for the time when the neuron output value passes through the ring structure, and calculate the wait time using the following formula: 41+ I + 2. H + O _REM + O _MOD . (8+ H +2. MaxPE ), in addition, when the unit synthesis number is sufficient for the output layer but insufficient for the hidden layer, wait for the time when the neuron output value passes through the ring structure, and calculate the waiting time using the following formula: 41+ I + H + O + H _REM + H _MOD . (8+ I +2. MaxPE ), if the synthesized number of the arithmetic unit is not enough for the hidden layer and the output layer, wait for the time when the neuron output value passes through the ring structure, and calculate the waiting time using the following formula: 41+ I + H + H _REM + O _REM + H _MOD . (8+ I +2. MaxPE )+ O _MOD . (8+ H +2. MaxPE ); where I is the number of input layer nodes, H is the number of hidden layer nodes, O is the number of output layer nodes, MaxPE is the number of arithmetic unit synthesis, and H _MOD is the hidden layer divided by the number of arithmetic units. The quotient, H _REM is the remainder of the number of combinations of the hidden layer in the hidden layer, O _MOD is the quotient of the number of synthesized elements of the output layer, and O _REM is the remainder of the number of combinations of the output layer divided by the arithmetic unit.