Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
  • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks

Definitions

• These inventions relate to computers and computer systems and particularly to virtual neurocomputer architectures for neural networks.
• LEARNING MACHINE SCALABLE FLOW VIRTUAL LEARNING NEUROCOMPUTER G. G. Pechanek,
• PLAN PYRAMID LEARNING ARCHITECTURE NEUROCOMPUTER, G. G.
• the neurons modeled on the neural processor are simulated in either a "direct” or a "virtual” implementation.
• each neuron would have a physical processing element (PE) available which could operate simultaneously in parallel with the other neuron PE's active in the system.
• PE physical processing element
• multiple neurons are assigned to individual hardware processing elements (PEs), requiring that the PE's processing be shared across its “virtual” neurons.
• the performance of the network will be greater under the "direct” approach, but due to the large number of neurons utilized in many network simulations and technology constraints limiting the number of "direct” neuron PEs which can be implemented, many neurocomputer designs utilize the "virtual" neurons concept to simulate more neurons than are available directly.
• the basic concept involved in a virtual neurocomputer is to provide some degree of parallelism, if possible, and then to divide the total number of neurons to be implemented in a network among the parallel processing elements available and to use time division multiplexing per physical processing element. This naturally spreads a single time unit out to cover the processing required for the number of virtual neural nodes assigned to a single physical PE.
• a virtual neural node represents one neuron in the network being simulated.
• a uni-processor can be used but it must handle all processing for all neural nodes in a sequential manner. Because of this, many virtual neurocomputers use a parallel arrangement of microprocessors. Parallel arrangements for neurocomputing may be reviewed. See Hecht-Nielsen 90: Savely IEEE 87 and Treleaven 89.
• the structure used usually allows for floating point hardware accelerators to be added for improved performance of each neural node calculation.
• An efficient communications network between the physical PE's is also required among the parallel processing elements to improve performance.
• For these virtual neurocomputers to function there must be local memory for the physical processors containing the network interconnection structure, weight matrix, and virtual PE activation state memory.
• our virtual system is scalable and applicable to networks and permits our improved hardware, illustrated in other applications, to be used where the physical hardware does not have enough neurons to achieve the results of the desired application.
• the new apparatus which acts as a virtual scalable neural array processor apparatus has an N neuron structure having weight multiplication units containing a multiplier, weight storage, and neuron output value storage for V neurons where V > N and V is the number of neurons in the neural network to be emulated on the N neuron structure.
• the first architectural preferred embodiment is the approach to be discussed. It modifies the SNAP structure to allow the sharing of the physical synapse processing elements such that a neuron input is not represented by a column or row of N synapse processors but by multiple rows or columns of synapse processors.
• a second architectural preferred embodiment to be discussed is the approach for TSNAP which replicates groups of synapse processing elements until sufficient synapse processing elements are available to cover the V neurons to be modeled while keeping N activation function generators (sigmoid generators).
• a third architectural preferred embodiment to be discussed is the general approach, representing a common approach for either SNAP or TSNAP, which adds an iterative adder with a partial summation storage element to each of the inputs of the N sigmoid generators and increases the weight storage appropriately at each synapse processor.
• FIGURE 1 illustrates the Virtual Neurocomputer.
• FIGURE 2 shows a SNAP-V1 with eight Virtual Neurons.
• FIGURE 3 illustrates a SNAP-V1 Bypass Adder.
• FIGURE 4 illustrates a SNAP tag matching multiplier
• FIGURE 5 shows SNAP-shots 1-4.
• FIGURE 6 shows SNAP-shot 5.
• FIGURE 7 shows SNAP-shots 6 through 9.
• FIGURE 8 shows SNAP-shot 10.
• FIGURE 9 illustrates SNAP-V2 with eight Virtual Neurons.
• FIGURE 10 illustrates a four Neuron T-SNAP.
• FIGURE 11 shows an 9-Neuron T-SNAP weight matrix with Neuron Y values.
• FIGURE 12 shows an eight Neuron Virtual T-SNAP with 4 physical Neurons.
• FIGURE 13 shows a 16-Neuron T-SNAP weight matrix with Neuron Y values.
• FIGURE 14 shows a 16-Neuron Virtual T-SNAP, and the Step 1 calculation of Y1 ', Y2', Y3', and Y4'.
• FIGURE 15 shows a 16-Neuron Virtual T-SNAP, and the Step 2 calculation of Y5', Y6', Y7', and Y8'.
• FIGURE 16 shows a 16-Neuron Virtual T-SNAP, and the Step 3 calculation of Y9', YA', YB', and YC.
• FIGURE 17 shows a 16-Neuron Virtual TSNAP, with four triangular sections and a Step 4 calculation of UD', YE', YF' and YG'.
• FIGURE 18 shows a 4-Neuron TSNAP-V2 with 16 Neuron weights
• FIGURE 19 shows a Virtual Architecture Comparison Summary.
• FIGURES may be separated in parts and as a convention we place the top of the FIGURE as the first sheet, with subsequent sheets proceeding down and across when viewing the FIGURE, in the event that multiple sheets are used.
• FIG. 1 The characteristics of a virtual neurocomputer are shown in Figure 1. Reference here may be had to a related system from Hecht-Nielsen illustrated by Sou'cek 88.
• Sou'cek 88 There are up to R physical processing elements depicted in Figure 1.
• each physical processor would be assigned the neural node calculation task for K neurons. This task is the calculation of the sum of products (weight ⁇ connected neuron output), the activation function, and possibly an output function.
• the parallel broadcast bus is used to communicate network interconnection information, neuron activation state values, etc. between the physical processors. In the structure of Figure 1 , the parallel bus is controlled by the system controller to ensure no bus conflicts. After the virtual neuron activation output values are calculated, the local system copies for these values must be updated in each neural processing element.
• the number V will be used to denote the number of neurons contained in the network to be modeled on the neurocomputer; the number N will denote the number physical neurons available in the physical implementation.
• V For virtual processing V > N.
• N and V are powers of 2.
• the computational tasks to be implemented by the neurocomputer architectures described herein are given by equation 1 and 2 which are based on a subset of the full Parallel Distributed Processing model. See here Rumelhart 86 and the Hopfield network illustrated by Hopfield 84.
• V is the number of neurons in the neural network.
• Y 1 is the j th neuron output value connected to the i th neuron input through a connection weight of W ij .
• F(z) is the neuron activation function which many times is set equal to a sigmoid activation function whose form, for example is:
• T is a global control parameter used to modify the slope of the sigmoid function for a given set of z values.
• Equations 1 and 2 for a completely connected V neuron network, contain four basic operations:
• the HOST computer assumes the responsibilities of initializing the network architectures.
• the HOST computer will be responsible for the loading of the number of neurons in the network to be simulated by the architecture, all the connection weights, the initial neuron values, the number of network update cycles to be run, as well as starting the model into execution.
• the HOST computer is also provided with the ability to read the neuron values at the completion of network execution.
• the initialization time and the Host processing time are considered as a separate issue. Only the performance during execution will be considered.
• the first architectural preferred embodiment is the approach to be discussed. It modifies the SNAP structure (see Vassiiiadis SNAP 90) to allow the sharing of the physical synapse processing elements such that a neuron input is not represented by a column or row of N synapse processors but rather by multiple rows or columns of synapse processors.
• a second architectural preferred embodiment to be discussed is the approach for TSNAP (see Pechanek T-SNAP) which replicates groups of synapse processing elements until sufficient synapse processing elements are available to cover the V neurons to be modeled while keeping N activation function generators (sigmoid generators)
• a third architectural preferred embodiment to be discussed is the general approach, representing a common approach for either SNAP or TSNAP, which adds an iterative adder with a partial summation storage element to each of the inputs of the N sigmoid generators and increases the weight storage appropriately at each synapse processor.
• the performance of each virtual neurocomputer is discussed.
• a summary compares each virtual architecture approach in terms of implementation costs and performance. The "best" architecture for a given application will depend upon these costs and will be chosen for the particular application.
• the system defined clock period is C, with all delays specified as multiples of C.
• the number of stages in the communicating adder tree is log 2 N. where N is the total number of physical neurons.
• This sequence of events requires a simple control mechanism such as the use of a counter whose output value is compared against delay values representing the listed events, namely: the multiplier delay, the log 2 N communicating adder tree - add mode delay, the sigmoid delay, and the log 2 N communicating adder tree - communications mode delay.
• SNAP-V1 SNAP VIRTUAL ARCHITECTURE 1
• SNAP-V1 The first approach to be described, termed SNAP-V1 , restricts V to a maximum of N 2 . Greater than N 2 neurons can be simulated with this architecture but a more involved control process would be required and this expanded capability will not be discussed in this description.
• SNAP-V1 the SNAP neuron definition (Vassiiiadis SNAP 90) is changed to accommodate the greater number of weighted inputs required per neuron.
• Figure 2 shows a virtual eight neuron network modeled on a 4 neuron SNAP. Sufficient internal storage for the weights and Y values is assumed to be available within the multiplier cell structures.
• a variation of the communicating adder to be used in the adder tree is required.
• This variation shown in Figure 3 SNAP-V1 Bypass Adder, allows the adder to be bypassed in a forward direction as well as the reverse communication mode direction.
• the various bypass paths are indicated by an arrow in Figure 3, and this symbology is also utilized in Figure 2 to indicate where the bypass adders are used.
• Table 1 depicts the driver controls, DR1 , DR2, DR3, DR4, and DR5, which are driven from a central common tree controller point, encoded from D3, D2, and D1 control signals.
• the sigmoid generators In order to ensure the neuron values are reverse communicated to the proper neuron input, the sigmoid generators must also tag the generated value. This tag accompanies the neuron value in its return path through the adder trees.
• the multiplier cell Y value input register will then require a neuron Y value tag matching compare function. This is depicted in Figure 4 where the loading of the Y value register is under control of the tag matching function. Two Y value registers are required, an old value and a new value.
• SNAP-V2 SNAP VIRTUAL ARCHITECTURE 2
• the total number of weight registers in terms of K is:
• TSNAP-V2 period K ⁇ K ⁇ M + (K + 1 )( log 2 N)C + ⁇ 1A + ⁇ s ) (4) TSNAP-V1 : TSNAP VIRTUAL ARCHITECTURE 1
• a virtual implementation on the TSNAP is our normally preferred embodiment, implementing the functions achieved by T-SNAP. This will be demonstrated utilizing a 4, 8, and a 16 neuron network modeled on a 4 neuron TSNAP.
• Figure 10 depicts the 4 neuron T-SNAP.
• the function F for all equations is assumed to be the sigmoid function.
• the equation for Neuron 3 is printed here for easy reference with Figure 10.
• Y 3 F(W 31 Y 1 + W 32 Y 2 + W 33 Y 3 + W 34 Y 4 )
• S represents the sigmoid generators.
• Figure 11 represents the 8 neuron weights and Y values required when modeled on an 8 neuron TSNAP.
• Figure 12 modifies the TSNAP structure of Figure 10 by replicating the diagonal cells, G-Cells, and addition trees and producing the partial summation on the diagonal of the structure. The summation is shown on the diagonal to demonstrate that the summation tree structure can be placed as required.
• the triangular sections replicated exclude the sigmoid generator though the size of the section is based on the number of sigmoid generators which is equal to the number of physical neurons N of the TSNAP section. For TSNAP-V1 it will be assumed that the triangular sections are replicated enough times to handle the largest network to be modeled on the system. The number of replicated triangular sections is given by:
• Y 3 F(W 31 Y 1 + W 32 Y 2 + W 33 Y 3 + W 34 Y 4 + W 35 Y 5 + W 36 Y 6 + W 37 Y 7 + W 38 Y 8 )
• ⁇ 7 F(W 71 Y 1 + W 72 Y 2 + W 73 Y 3 + W 74 Y 4 + W 75 Y 5 + W 76 Y 6 + W 77 Y 7 + W 78 Y 8 )
• the neuron values would be tagged such that only the correct neuron values are loaded into the proper triangular section.
• the compare of the tag can be done prior to sending the neuron value into a triangular section or at each Y value storage register.
• Figure 12 One of the advantages of these illustrated embodiments which we prefer is that the basic concept shown in Figure 12 can be continued for larger networks by replication of the multipliers and communicating adder trees while keeping the same number of neuron activation function generators.
• This approach for a 16 neuron network is shown in Figure 14, 15, 16, and 17.
• Figures 14, 15, 16, and 17 For reference all the weights and neuron values for the 16 neurons, as would be used in a 16 neuron T-SNAP, are shown in Figure 13.
• Figures 14, 15, 16, and 17 contain 4 replicated T-SNAP sections.
• the weights and output values for neurons 1 to 4 are depicted in Figure 14 STEP 1.
• Figure 15 STEP 2 shows the weights and structure required for neurons 5 to 8.
• Figure 16 STEP 3 shows neurons 9 to 12
• Figure 17 STEP 4 shows neurons 13 to 16.
• Y 3 F(W 3,1 Y 1 + W 3,2 Y 2 + W 3,3 Y 3 + W 3,4 Y 4 + W 3,5 Y 5 + W 3,6 Y 6 + W 3 ,7 Y 7 + W 3,8 Y 8 ) +
• Y 7 F(W 7,1 Y 1 + W 7,2 Y 2 + W 7,3 Y 3 + W 7,4 Y 4 + W 7,5 Y 5 + W 7,6 Y 6 + W 7,7 Y 7 + W 7,8 Y 8 ) + (W 7,9 Y 9 + W 7,10 Y 10 + W 7,11 Y 11 + W 7, 12 Y 12 + W 7,13 Y 13 + W 7,14 Y 14 + W 7,15 Y 15 + W 7,16 Y 16 )
• Y 12 F(W 12, 1 Y 1 + W 12,2 Y 2 + W 12,3 Y 3 + W 12,4 Y 4 + W 12,5 Y 5 + W 12,6 Y 6 + W 12,7 Y 7 + W 12,8 Y 8 ) + (W 12,9 Y 9 + W 12,10 Y 10 + W 12,11 Y 11 + W 12, 12 Y 12 + W 12,13 Y 13 + W 12,14 Y 14 + W 12,15 Y 15 + W 12,16 Y 16 )
• Y 14 F(W 14,1 Y 1 + W 14.2 Y 2 + W 14,3 Y 3 + W 14,4 Y 4 + W 14.5 Y 5 + W 14,6 Y 6 + W 14,7 Y 7 + W 14,8 Y 8 ) +
• S represents the sigmoid generators.
• TSNAP - V1 period K( ⁇ M + 2( log 2 N + log 2 K) ⁇ A + ⁇ s )
• TSNAP-V2 TSNAP VIRTUAL ARCHITECTURE 2
• the TSNAP-V2 performance is:
• TSNAP - V2 period K(K ⁇ M + 2( log 2 N ⁇ A + ⁇ IA ) + ⁇ s )

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computing Systems (AREA)
• Software Systems (AREA)
• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Biomedical Technology (AREA)
• Biophysics (AREA)
• General Health & Medical Sciences (AREA)
• Mathematical Physics (AREA)
• Evolutionary Computation (AREA)
• Computational Linguistics (AREA)
• Molecular Biology (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Artificial Intelligence (AREA)
• Neurology (AREA)
• Feedback Control In General (AREA)
• Multi Processors (AREA)
• Devices For Executing Special Programs (AREA)
• Complex Calculations (AREA)
• Executing Machine-Instructions (AREA)
• Advance Control (AREA)

Applications Claiming Priority (4)

Publications (1)

Family

ID=27062243

Family Applications (4)

Family Applications Before (2)

Family Applications After (1)

Country Status (4)

Families Citing this family (36)

Family Cites Families (14)

• 1991
  • 1991-05-17 JP JP3510421A patent/JP2746350B2/ja not_active Expired - Fee Related
  • 1991-05-17 WO PCT/US1991/003315 patent/WO1991018349A1/en not_active Application Discontinuation
  • 1991-05-17 WO PCT/US1991/003316 patent/WO1991018350A1/en not_active Application Discontinuation
  • 1991-05-17 JP JP3517778A patent/JP2663996B2/ja not_active Expired - Fee Related
  • 1991-05-17 JP JP3509437A patent/JP2663995B2/ja not_active Expired - Fee Related
  • 1991-05-17 EP EP91910713A patent/EP0484506A1/en not_active Withdrawn
  • 1991-05-17 EP EP91910047A patent/EP0486635A1/en not_active Withdrawn
  • 1991-05-17 EP EP91918394A patent/EP0486684A1/en not_active Withdrawn
  • 1991-05-17 EP EP91920989A patent/EP0484522A1/en not_active Withdrawn
  • 1991-05-17 JP JP3510818A patent/JP2502867B2/ja not_active Expired - Lifetime
  • 1991-05-17 WO PCT/US1991/003318 patent/WO1992001257A1/en not_active Application Discontinuation
  • 1991-05-17 WO PCT/US1991/003317 patent/WO1991018351A1/en not_active Application Discontinuation
• 1994
  • 1994-04-22 US US08/231,853 patent/US5509106A/en not_active Expired - Fee Related
  • 1994-12-22 US US08/362,087 patent/US5542026A/en not_active Expired - Fee Related
• 1995
  • 1995-06-02 US US08/459,191 patent/US5617512A/en not_active Expired - Fee Related

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events