TW201741943A - Deep learning neural network classifier using non-volatile memory array - Google Patents

Abstract

An artificial neural network device that utilizes one or more non-volatile memory arrays as the synapses. The synapses are configured to receive inputs and to generate therefrom outputs. Neurons are configured to receive the outputs. The synapses include a plurality of memory cells, wherein each of the memory cells includes spaced apart source and drain regions formed in a semiconductor substrate with a channel region extending there between, a floating gate disposed over and insulated from a first portion of the channel region and a non-floating gate disposed over and insulated from a second portion of the channel region. Each of the plurality of memory cells is configured to store a weight value corresponding to a number of electrons on the floating gate. The plurality of memory cells are configured to multiply the inputs by the stored weight values to generate the outputs.

Description

Deep learning neural network classifier using non-volatile memory array [Reciprocal Reference of Related Applications]

The present application claims the benefit of U.S. Provisional Application Serial No. 62/337,760, filed on Jan. 17, s.

The present invention relates to neural networks.

Artificial neural networks simulate biological neural networks (the animal's central nervous system, particularly the brain) that are used to estimate or approximate functions that may depend on a large number of inputs and are generally unknown. Artificial neural networks typically include an interconnected "neuron" layer that exchanges information between each other. Figure 1 illustrates an artificial neural network in which a circle represents an input or layer of a neuron. Connections (called synapses) are represented by arrows and have digital weights that can be tuned based on experience. This allows the neural network to be adaptively input and able to learn. In general, a neural network includes multiple input layers. There is typically one or more intermediate layers of neurons, and a neuron output layer that provides an output of the neural network. The quasi-neurons make decisions individually or collectively based on data received from synapses.

One of the major challenges in the development of artificial neural networks for high performance information processing is the lack of appropriate hardware technology. In fact, the actual neural network relies on a very large number of synapses, thereby achieving high connectivity between neurons, that is, extremely high computational parallelism. In principle, this complexity can be achieved with a digital supercomputer or a dedicated graphics processing unit set. The group came to an end. However, in addition to high costs, these approaches also suffer from inferior energy efficiencies compared to bio-networks because they perform low-precision analog calculations and therefore consume substantially less energy. CMOS analog circuits have been used in artificial neural networks, but most CMOS implemented synapses are too large for a given high number of neurons and synapses.

The foregoing problems and need to be addressed by using one or more non-volatile memory arrays as synaptic artificial neural network devices. The neural network device includes: a first plurality of synapses configured to receive a first plurality of inputs, and a first plurality of outputs from the first plurality of inputs; and a first plurality of neurons, The first plurality of outputs are configured to receive. The first plurality of synapses includes: a plurality of memory cells, wherein each of the memory cells includes a spaced apart source region and a drain region formed in a semiconductor substrate, the channel region being in the source region Extending from the drain region; a floating gate disposed above and insulated from the first portion of the channel region; and a non-floating gate disposed above the second portion of the channel region and The second part is insulated. Each of the plurality of memory cells is configured to store a weight value corresponding to a plurality of electrons on the floating gate. The plurality of memory cells are configured to multiply the first plurality of inputs by the stored weight values to generate the first plurality of outputs.

Other objects and features of the present invention will become apparent from the description and appended claims.

10‧‧‧ memory unit

12‧‧‧Semiconductor substrate/substrate

14‧‧‧Source Zone/Source

14a‧‧‧Source line/horizontal source line

16‧‧‧Bungee Area/Bungee

16a‧‧‧ bit line

16a1‧‧‧first bit line

16a2‧‧‧second bit line

18‧‧‧Channel area

20‧‧‧Float

22‧‧‧Control gate

22b‧‧‧Part II

22a‧‧‧Part 1 / Control gate / level control gate / control gate

22a1‧‧‧First control gate

22a2‧‧‧Second control gate

24‧‧‧Intermediate insulator

26‧‧‧gate oxide

28‧‧‧Selection gate

28a‧‧‧ horizontal selection gate line

28a1‧‧‧Selected brake line

28a2‧‧‧Selecting the brake line

30‧‧‧ wipe the gate

30a‧‧‧Erase the brake line

31‧‧‧Digital-to-analog converter

32, 32a~e‧‧‧VMM

33‧‧‧Array/memory array of non-volatile memory cells

34‧‧‧Erase gate and word line gate decoder

35‧‧‧Control gate decoder

36‧‧‧ bit line decoder

37‧‧‧Source Line Decoder

38‧‧‧Differential summing operational amplifier/suming operational amplifier

39‧‧‧Activate the function circuit

50‧‧‧current-voltage logarithmic converter

52‧‧‧Voltage-current logarithmic converter

54‧‧‧Refer to Gnd's current summer

56‧‧‧Refer to Vdd's current summer

C1‧‧‧Characteristic map

C2‧‧‧Characteristic map

C3‧‧‧Characteristic map

CB1‧‧‧ synapse

CB2‧‧‧ synapse

CB3‧‧‧ synapse

CB4‧‧‧ synapse

CG‧‧‧Control gate

D‧‧‧Shared bungee area

DAC 40‧‧‧Digital-to-Analog Converter

EG‧‧‧wipe the brake line

IComp 44‧‧‧ Current Comparator

Iin‧‧‧Incoming current/input current

Iin0‧‧‧Incoming current / input current

Iin1‧‧‧Incoming current/input current

Iin2‧‧‧Incoming current/input current

Iin3‧‧‧Incoming current/input current

Iin4‧‧‧Incoming current / input current

Iin5‧‧‧Incoming current/input current

Iin7‧‧‧Incoming current/input current

Iout‧‧‧Output current / output / current

Iout0‧‧‧ matrix output

Iout1‧‧‧ matrix output

Iout2‧‧‧ matrix output

Iout3‧‧‧ matrix output

Iout4‧‧‧ matrix output

P1‧‧‧ activation function

P2‧‧‧ activation function

S‧‧‧shared source area

S1‧‧‧ feature map

S2‧‧‧ feature map

S3‧‧‧ output

V/I Conv 42‧‧‧Voltage-to-Current Converter

V/I Conv 48‧‧‧Voltage-to-Current Converter

VComp 46‧‧‧Voltage Comparator

Vin‧‧‧Input voltage

Vin0‧‧‧Matrix Input/Input

Vin1‧‧‧ matrix input

Vin2‧‧‧ Matrix Input

Vin3‧‧‧ Matrix Input

Vin4‧‧‧ Matrix Input

Vin5‧‧‧ Matrix Input

Vin6‧‧‧ Matrix Input

Vin7‧‧‧ matrix input

WL‧‧‧Selected brake line

FIG. 1 is a diagram illustrating an artificial neural network.

2 is a side cross-sectional view of a conventional 2-gate non-volatile memory cell.

3 is a diagram illustrating a conventional array architecture of the memory cell of FIG. 2.

4 is a side cross-sectional view of a conventional 2-gate non-volatile memory unit.

FIG. 5 is a diagram illustrating a conventional array architecture of the memory cell of FIG. 4.

Figure 6 is a side cross-sectional view of a conventional 4-gate non-volatile memory unit.

7 is a diagram illustrating a conventional array architecture of the memory cell of FIG. 6.

Figure 8A is a diagram illustrating a uniformly spaced neural network weight level assignment.

Figure 8B is a diagram illustrating neural network weight level assignments with uneven spacing.

Figure 9 is a flow chart illustrating a two-way tuning algorithm.

FIG. 10 is a block diagram illustrating weight mapping using current comparison.

FIG. 11 is a block diagram illustrating weight mapping using voltage comparison.

Figure 12 is a diagram illustrating different levels of an exemplary neural network using a non-volatile memory array.

Figure 13 is a block diagram illustrating a vector multiplier matrix.

Figure 14 is a block diagram illustrating various levels of a vector multiplier matrix.

15 to 16 are schematic views illustrating a first architecture of an array of four-gate memory cells.

17 to 18 are schematic views illustrating a second architecture of an array of four-gate memory cells.

19 is a schematic diagram illustrating a third architecture of an array of four-gate memory cells.

20 is a schematic diagram illustrating a fourth architecture of an array of four-gate memory cells.

21 is a schematic diagram illustrating a fifth architecture of an array of four-gate memory cells.

Figure 22 is a schematic diagram illustrating a sixth architecture of an array of four-gate memory cells.

23 is a schematic diagram illustrating a first architecture of an array of two-gate memory cells.

24 is a schematic diagram illustrating a second architecture of an array of two-gate memory cells.

Fig. 25 is a diagram illustrating a current-voltage logarithmic converter.

Fig. 26 is a diagram illustrating a voltage-current logarithmic converter.

Figure 27 is a diagram illustrating a current summer with reference to Gnd.

28 is a diagram illustrating a current summer with reference to Vdd.

29 is a diagram illustrating the use of N ² neural network inputs of a non-volatile memory array.

Figure 30 is a diagram illustrating the use of N ² neural network inputs of a non-volatile memory array.

31 is a diagram illustrating the use of a neural network input of a non-volatile memory array having periodically shifted input lines.

32 is a schematic diagram of a memory array architecture illustrating the input line of FIG. 15 but with periodic shifts.

33 is a schematic diagram of a memory array architecture illustrating the input line of FIG. 20 but having periodic shifts.

The artificial neural network of the present invention utilizes a combination of CMOS technology and a non-volatile memory array. Digital non-volatile memory systems are well known. An array of discrete gate non-volatile memory cells is disclosed, for example, in U.S. Patent No. 5,029,130 ("the '130 patent"). The memory cell system is shown in Figure 2. Each memory cell 10 includes a source region 14 formed in a semiconductor substrate 12 and The drain region 16, the source region 14 and the drain region 16 have a channel region 18. A float gate 20 is formed over and insulated from the first portion of the channel region 18 (and controls the conductivity of the first portion) and over a portion of the drain region 16. A control gate 22 has a first portion 22a and a second portion 22b. The first portion 22a is disposed above and insulated from the second portion of the channel region 18 (and controls the conductivity of the second portion). The two portions 22b extend upwardly above the float gate 20. The float gate 20 and the control gate 22 are insulated from the substrate 12 by a gate oxide 26.

Erasing the memory cell (where electrons are removed from the floating gate) by placing a high positive voltage on the control gate 22 causes the electrons on the floating gate 20 to tunnel through the Fowler-Nordheim tunnel from the float gate 20 Through the intermediate insulator 24 to the control gate 22.

The memory cell is programmed (where the electrons are placed on the floating gate) by placing a positive voltage on the control gate 22 and a positive voltage on the drain 16. The electron flow will flow from the source 14 towards the drain 16 . When electrons reach the gap between the control gate 22 and the float gate 20, the electrons will accelerate and become hot. Some of the changed hot electrons will be injected into the floating gate 20 through the gate oxide 26 due to the electrostatic attraction from the floating gate 20.

The memory cell (which turns on the channel region under the control gate) is read by placing the positive read voltage on the drain 16 and the control gate 22. If the floating gate 20 is positively charged (i.e., the electrons are erased and positively coupled to the drain 16), the portion of the channel region below the floating gate 20 is also turned on, and current will flow across the channel region 18, which senses Is erased or "1" status. If the floating gate 20 is negatively charged (i.e., electronically programmed), the portion of the channel region below the floating gate 20 is mostly or completely disconnected, and current will not flow across the channel region 18 (or there will be a slight flow) The system senses a stylized or "0" state.

The architecture of the memory array is shown in Figure 3. Memory unit 10 Set into columns and rows. In each row, the memory cells are arranged end-to-end in a mirror image manner such that they are formed into a pair of memory cells, each pair sharing a common source region 14(S), and each group of adjacent memories The body unit pairs share a common bungee region 16(D). All of the source regions 14 for any given column of memory cells are electrically connected together by a source line 14a. All of the drain regions 16 for any given row of memory cells are electrically connected together by a single bit line 16a. All of the control gates 22 for any given array of memory cells are electrically connected together by a control gate 22a. Therefore, although the memory cells can be individually programmed and read, the erase of the memory cells is performed in columns and columns (the columns of memory cells are wiped together by applying a high voltage on the control gate 22a). except). If a particular memory cell is to be erased, all memory cells in the same column are also erased.

One of ordinary skill in the art understands that the source and drain are interchangeable, with the floating gate extending partially above the source rather than the drain, as shown in FIG. FIG. 5 best illustrates a corresponding memory cell architecture including a memory cell 10, a source line 14a, a bit line 16a, and a control gate 22a. As is apparent from the figures, memory cells 10 of the same column share the same source line 14a and the same control gate 22a, while the drain regions of all cells of the same row are electrically connected to the same bit line 16a. . The array design is optimized for digital applications and is allowed, for example, by applying 1.6V and 7.6V to selected control gates 22a and source lines 14a, respectively, and grounding the selected bit lines 16a. Select individual stylization of the unit. The non-selected memory cells of the same pair are avoided by applying a voltage greater than 2 volts across the unselected bit line 16a and grounding the remaining lines. The memory unit 10 cannot be erased individually because the program responsible for erasing (electronic Fowler-Nordheim tunneling from the float gate 20 to the control gate 22) is only subjected to the drain voltage (i.e., the same source is shared in the column direction). Unique to two adjacent cells of pole line 14a Can be a weak effect of different voltages).

Separate gate memory cells having more than two gates are also known. For example, having a source region 14, a drain region 16, a floating gate 20 above a first portion of the channel region 18, a selection gate 28 above a second portion of the channel region 18, and a gate above the floating gate 20 A control unit 22, and a memory unit of a wiper 30 above the source region 14, are known, as shown in Figure 6 (see, e.g., U.S. Patent No. 6,747,310, incorporated herein by reference herein In this article). Here, except for the float 20, all gates are non-floating, meaning that they are electrically connected or connectable to a voltage source. The stylization is displayed by injecting the heated electrons into the floating gate 20 from the channel region 18. The erase is indicated by electron tunneling from the floating gate 20 to the erase gate 30.

The architecture for a four-gate memory cell array can be configured as shown in Figure 7. In this embodiment, each horizontal select gate 28a electrically connects all of the select gates 28 for the other memory cells. Each of the horizontal control gates 22a electrically connects all of the control gates 22 for the other memory cells. Each of the horizontal source lines 14a electrically connects all of the source regions 14 of the two columns of memory cells for sharing the source regions 14. Each of the bit lines 16a electrically connects all of the drain regions 16 for the memory cells of the row. Each erase gate 30a electrically connects all of the erase gates 30 for sharing the two columns of memory cells of the erase gate 30. As with the previous architecture, individual memory cells can be programmed and read independently. However, it is not possible to erase the unit individually. The erasing is performed by placing a high positive voltage on the erase gate line 30a, which results in simultaneous erasing of the two columns of memory cells sharing the same erase gate line 30a. An exemplary operating voltage can be included in Table 1 below (in this embodiment, select gate 28a can be referred to as word line WL):

Two modifications were made to use the above non-volatile memory array in the neural network. First, the lines are reconfigured such that each memory unit can be individually programmed, erased, and read without adversely affecting the memory state of other memory cells in the array, as explained further below. . Second, provide continuous (analog) stylization of memory cells. In particular, the memory state of each memory cell in the array (ie, the charge on the floating gate) can be continuously changed from the fully erased state to the independent and in the case of minimal interference to other memory cells. Fully stylized state and vice versa. This means that the unit storage is analogous or at least can store at least one of a number of discrete values, thereby allowing extremely accurate and individual tuning of all of the units in the memory array, and for making and storing synaptic weights for the neural network. Fine-tuning the idea of memory arrays.

Memory unit stylization and storage

The neural network weight level assignments as stored in the memory unit may be evenly spaced as shown in Figure 8A, or unevenly spaced as shown in Figure 8B. Stylization of non-volatile memory cells can be implemented using a two-way tuning algorithm such as that shown in FIG. Icell is the read current of the programmed target cell, and Itarget is the desired read current when the cell is ideally programmed. The read target cell reads the current Icell (step 1) and compares it with the target read current Itarget (step 2). If the target unit reads When the current Icell is greater than the target read current Itarget, a programmed tuning process (step 3) is performed to increase the number of electrons on the floating gate (where the lookup table is used to determine the desired programmed voltage VCG on the control gate) (step 3a to 3b), which can be repeated as needed (step 3c). If the target cell read current Icell is smaller than the target read current Itarget, an erase tuning process (step 4) is performed to reduce the number of electrons on the floating gate (where the lookup table is used to determine the desired erase voltage VEG on the erase gate) (Steps 4a to 4b), which can be repeated as needed (Step 4c). If the programmed tuning process exceeds the target read current, an erase tuning process is performed (step 3d and begins with step 4a) and vice versa (step 4d and begins with step 3a) until (within an acceptable delta value) The target reads the current.

In contrast, stylization of non-volatile memory cells can be implemented using a one-way tuning algorithm that utilizes programmatic tuning. With this algorithm, the memory cells are initially completely erased, and then the stylized tuning steps 3a through 3c in Figure 9 are performed until the read current of the target cell reaches the target threshold. Alternately, tuning of the non-volatile memory cells can be implemented using a one-way tuning algorithm that utilizes erase tuning. In this approach, the memory cells are initially fully programmed, and then the erase tuning steps 4a through 4c in Figure 9 are performed until the read current of the target cell reaches the target threshold.

FIG. 10 is a diagram illustrating weight mapping using current comparison. The weight digits (eg, the 5-bit weight for each synapse, representing the target digit weight for the memory unit) are input to a digital-to-analog converter (DAC) 40, thereby converting the bits into Voltage Vout (eg, 64 voltage level - 5 bits). Vout is converted to a current Iout by a voltage-to-current converter V/I Conv 42 (eg, 64 current level - 5 bits). The current is supplied to the current comparator IComp 44. Stylized or erased algorithm enable input to remember Memory unit 10 (eg, erase: increments EG voltage; or stylize: increments CG voltage). The memory cell output current Icellout (ie, from a read operation) is supplied to the current comparator IComp 44. The current comparator IComp 44 compares the memory cell current Icellout with the current Iout derived from the weighting digit bit to generate a signal indicative of the weight stored in the memory unit 10.

FIG. 11 is a diagram illustrating weight mapping using voltage comparison. The weight digits (eg, the 5-bit weight for each synapse) are input to a digital-to-analog converter (DAC) 40, thereby converting the bits to a voltage Vout (eg, 64 voltage level -5) Bit). Vout is supplied to the voltage comparator VComp 46. The stylized or erased algorithm enables input to the memory unit 10 (eg, erase: increments the EG voltage; or stylizes: increments the CG voltage). The memory cell output current Icellout is supplied to the current-to-voltage converter I/V Conv 48 for conversion to a voltage V2out (eg, 64 voltage level - 5 bits). The voltage V2out is supplied to the voltage comparator VComp 46. Voltage comparator VComp 46 compares voltage Vout with V2out to generate a signal indicative of the weight stored in memory unit 10.

Neural network using a non-volatile memory cell array

Figure 12 conceptually illustrates a non-limiting example of a neural network using a non-volatile memory array. This example uses a non-volatile memory array neural network for facial recognition applications, but any other suitable application can be implemented using a neural network based on a non-volatile memory array. S0 is an input, which is a 32×32 pixel RGB image with 5-bit precision for this example (ie, three 32×32 pixel arrays, one for each color R, G, and B, each pixel is 5 bits) Accuracy). Synaptic CB1 from S0 to C1 There are two different sets of weights and shared weights, and the input image is scanned using a 3 x 3 pixel overlay filter (core) to shift the filter by 1 pixel (or more than 1 pixel as specified by the model). Specifically, the synapse CB1 is provided with a value of 9 pixels (ie, referred to as a filter or a core) in a 3×3 portion of the image, whereby the nine input values are multiplied by an appropriate weight, and After summing the output of the product, a single output value is determined and provided by the first neuron of CB1 to produce a pixel of one of the feature map layers C1. The 3x3 filter is then shifted one pixel to the right (i.e., three pixel rows are added on the right side and three pixel rows are reduced on the left side), thereby providing this new positioning filter to synapse CB1. The 9 pixel values are thereby multiplied by the same weight and the second single output value is determined by the associated neuron. This process continues until the 3x3 filter scans all three colors and all bits (precision values) across the entire 32x32 pixel image. The process is then repeated using different sets of weights to produce different feature maps C1 until all feature map layers C1 have been calculated.

In this example, at C1, there are 16 feature maps each having 30 x 30 pixels. Each pixel is a new feature pixel that is self-multiplied with the input and core, and thus each feature map is a two-dimensional array, and thus in this example, the synapse CB1 constitutes a 16-layer two-dimensional array (note, reference is made herein) The neuron layer and array are logical relationships, not necessarily entity relationships, ie, the arrays are not necessarily oriented in a solid two-dimensional array). Each of the 16 feature maps is generated by sixteen different sets of synaptic weights applied to the filter scan. The C1 feature maps can all be directed to different aspects of the same image feature, such as boundary recognition. For example, a first map (generated using a first weight recombination, shared for all scans used to generate this first map) can identify a circular edge, and a second map (using a second weight different from the first weight recombination) The group produces) recognizable rectangular edges, or aspect ratios of certain features, and so on.

The activation function P1 (merging) is applied before C1 to S1, thereby merging values from successive, non-overlapping 2x2 regions in each feature map. The purpose of the merge phase is to average the nearby locations (or the maximum function can also be used), for example to reduce the correlation of edge locations and to reduce the data size before proceeding to the next phase. At S1, there are 16 15x15 feature maps (i.e., sixteen different arrays of 15 x 15 pixels each). The synapse and associated neurons in CB2 from S1 to C2 use a 4x4 filter, 1 pixel filter shift to scan the map in S1. At C2, there are 22 12 x 12 feature maps. The activation function P2 (merging) is applied before C2 to S2, thereby merging values from successive non-overlapping 2x2 regions in each feature map. At S2, there are 22 6 x 6 feature maps. An activation function is applied at synapses CB3 from S2 to C3, wherein each of C3 is connected to each of S2. At C3, there are 64 neurons. The synapse CB4 from C3 to output S3 fully connects S3 to C3. The output at S3 includes 10 neurons, with the highest output neuron determining the category. This output may, for example, indicate the identification and classification of the content of the original image.

Each of the synapses is implemented using an array of non-volatile memory cells, or a portion of an array. Figure 13 is a block diagram of a vector matrix multiplication (VMM) array that includes non-volatile memory cells and acts as a synapse between the input layer and the next layer. Specifically, the VMM 32 includes an array 33 of non-volatile memory cells, an erase gate and word line gate decoder 34, a control gate decoder 35, a bit line decoder 36, and a source line decoder 37, which The decoder decodes the input for the memory array 33. Source line decoder 37 also decodes the output of the memory cell array in this example. The memory array service serves two purposes. First, it stores the weights that will be used by the VMM. Second, the memory array effectively makes The input is multiplied by the weight stored in the memory array to produce an output that will be the input to the next layer or to the input of the final layer. By performing the multiply function, the memory array eliminates the need for a separate multiply logic circuit and is also power efficient.

The output of the memory array is supplied to a differential summing operational amplifier 38 which adds the outputs of the memory cell array to produce a single value for the convolution. The added output value is then supplied to the activation function circuit 39 of the correction output. The corrected output value causes the element of the feature map to be the next layer (such as C1 in the above description) and then applied to the next synapse to produce the next feature map layer or final layer. Thus, in this example, the memory array constitutes a plurality of synapses (which are received from a previous neuron layer or from an input layer such as an image database), and the summing operational amplifier 38 and the activation function circuit 39 constitute A plurality of neurons.

Figure 14 is a block diagram of various levels of the VMM. As shown in FIG. 14, the input is converted by digital-to-analog by digital-to-analog converter 31 and provided to input VMM 32a. The output produced by input VMM 32a is provided as an input to the next VMM (hidden level 1) 32b, thereby subsequently producing an output that is provided to the next VMM (hidden level 2) 32b, and the like. The various layers of VMM 32 function as different layers of synapses and neurons in the Convolutional Neural Network (CNN). Each VMM can be an independent non-volatile memory array, or multiple VMMs can use different portions of the same non-volatile memory array, or multiple VMMs can use overlapping portions of the same non-volatile memory array.

Figure 15 illustrates an array of quad gate memory cells configured as a drain sum matrix multiplier (i.e., such as shown in Figure 6). The various gate lines and zone lines used for the array of Figure 15 are the same as those of Figure 7 (where the same component numbers are used for the corresponding structures), Except that the erase gate line 30a operates vertically rather than horizontally (i.e., each erase gate 30a connects all erase gates 30 for the memory cells of the row) so that each memory unit 10 can Stylized, erased, and read independently. After staging each of the memory cells with the appropriate weight values for the cell, the array acts as a buck summation matrix multiplier. The matrix input is Vin0...Vin7 and placed on select gate line 28a. The matrix outputs Iout0...IoutN for the array of Fig. 15 are generated on the bit line 16a. For all cells in the row, each output Iout is the sum of the cell current I times the weight W stored in the cell: Iout=Σ(Iij*Wij)

Each memory unit (or pair of memory units) acts as a single synapse with a weight value that is defined by the sum of the weight values stored in the memory unit (or pair of memory units) in the other row. Output current Iout. The output of any given synapse is in the form of a current. Thus, each subsequent VMM stage after the first stage preferably includes circuitry for converting the incoming current from the previous VMM stage to the voltage to be used as the input voltage Vin. Figure 16 illustrates an example of such a current-to-voltage conversion circuit system that is a modified memory cell column that converts the incoming currents Iin0...IinN logarithmically into input voltage Vin0..VinN.

The memory cells described herein are biased in weak inversion, Ids = Io * e ^{(Vg - Vth) / kVt} = w * Io * e ^{(Vg) / kVt} for the use of memory cells to convert the input current into For an IV logarithmic converter of input voltage: Vg=k*Vt*log[Ids/wp*Io] For a memory array used as a vector matrix multiplier VMM, the output current is: Iout=wa * Io * e ^(Vg)/kVt , ie Iout=(wa/wp)* Iin=W * Iin

17 and 18 illustrate another configuration of an array of quad gate memory cells (i.e., such as shown in FIG. 6) configured as a drain sum matrix multiplier. The lines for the arrays of Figures 17 and 18 are the same as the ones of the arrays of Figures 15 and 16, except that the source line 14a operates vertically rather than horizontally (i.e., the source lines 14a will be used for each other). All of the source regions 14 of the row memory cells are connected together) and the erase gates 30a operate horizontally rather than vertically (i.e., each erase gate 30a will be used for all of the memory cell pairs) The erase gates 30 are connected together so that the memory cells can be independently programmed, erased, and read. The matrix inputs Vin0...VinN are held on the select gate line 28a, and the matrix outputs Iout0...IoutN are held on the bit line 16a.

Figure 19 illustrates another configuration of an array of quad gate memory cells (i.e., such as shown in Figure 6) configured as a gate coupling/source sum matrix multiplier. The lines for the array of Figure 19 are the same as those of Figures 15 and 16, except that the select gate 28a operates vertically and both of the select gates are used for each row of memory cells. Specifically, each row of memory cells includes two select gates: a first select gate 28a1 that connects all of the select gates 28 of the odd column memory cells; and a second select gate 28a2 that will have even columns All of the selection gates 28 of the memory unit are connected together.

The circuit at the top and bottom of Figure 19 is used to input current The Iin0...IinN logarithm is converted to the input voltage Vin0..VinN. The matrix inputs shown in this figure are Vin0...Vin5 and placed on select gates 28a1 and 28a2. Specifically, the input Vin0 is placed on the select line 28a1 for the odd cells in row 1. Vin1 is placed on select gate line 28a2 for the even units in row 1. Vin2 is placed on select gate line 28a1 for odd cells in row 2. Vin3 is placed on select gate line 28a2 for even units in row 2, and so on. The matrix outputs Iout0...Iout3 are provided on the source line 14a. Bit line 16a is biased at a fixed bias voltage VBLrd. For all cells in the memory cell, each output Iout is the sum of the cell current I times the weight W stored in the cell. Thus, for this architecture, each column of memory cells acts as a single synapse with a weight value represented as the output current Iout specified by the sum of the weight values stored in the memory cells in the column.

Figure 20 illustrates another configuration of an array of quad gate memory cells (i.e., such as shown in Figure 6) configured as a gate coupling/source sum matrix multiplier. The lines for the array of Figure 20 are the same as the one of Figure 19 except that bit line 16 runs vertically and both of the bit lines are used for each row of memory cells. Specifically, each row of memory cells includes two bit lines: a first bit line 16a1 that will connect all of the drain regions of adjacent pairs of memory cells (two memory cells share the same bit line contact) Connected together; and a second bit line 16a2 that connects all of the drain regions of the next adjacent pair of memory cells. The matrix inputs Vin0...VinN are held on the select gate lines 28a1 and 28a2, and the matrix outputs Iout0...IoutN are held on the source line 14a. All sets of first bit lines 16a1 are biased at an offset level, for example, 1.2v, and all sets of second bit lines 16a2 are biased at another offset level, eg, 0v. Source line 14a is biased at a virtual offset Quasi, for example, 0.6v. For each pair of memory cells sharing the shared source line 14a, the output current will be the differential output of the top cell minus the bottom cell. Therefore, each output Iout is the sum of these differential outputs: Iout=Σ(Iiju*Wiju-Iijd*Wijd)

SL voltage ~1⁄2 Vdd, ~0.5v Therefore, for this architecture, each pair of memory cells acts as a single synapse with a weight value, which is expressed as the output current Iout, which is stored in The sum of the differential outputs specified by the weight values in the memory cells in the pair of memory cells.

Figure 21 illustrates another configuration of an array of quad gate memory cells (i.e., such as shown in Figure 6) configured as a gate coupling/source sum matrix multiplier. The line for the array of Fig. 21 is the same as the one of Fig. 20 except that the erase gate 30a is operated horizontally, and the control gate 22a is operated vertically and both of the control gates are used for each row of memory cells. . Specifically, each row of memory cells includes two control gates: a first control gate 22a1 that connects all of the control gates 22a of the odd-numbered memory cells; and a second control gate 22a2 that has an even-numbered column All of the control gates 22a of the memory unit are connected together. The matrix inputs Vin0...VinN are held on the select gate lines 28a1 and 28a2, and the matrix outputs Iout0...IoutN are held on the source line 14a.

Figure 22 illustrates another configuration of an array of quad gate memory cells (i.e., such as shown in Figure 6) configured as a source sum matrix multiplier. The lines and inputs for the array of Figure 22 are the same as those of Figure 17. However, instead of providing the output on the bit line 16a, the outputs are provided on the source line 14a. Matrix input Vin0...VinN remains On the selection gate line 28a.

23 illustrates a configuration of an array of two-gate memory cells (ie, such as those shown in FIG. 1) configured as a drain sum matrix multiplier. The line for the array of Figure 23 is the same as the one of Figure 5 except that the horizontal source line 14a has been replaced with a vertical source line 14a. Specifically, each of the source lines 14a is connected to all of the source regions in the other memory cells. The matrix inputs Vin0...VinN are placed on the control gate 22a. Matrix outputs Iout0...IoutN are generated on bit line 16a. For all cells in a row, each output Iout is the sum of the cell current I times the weight W stored in the cell. Each row of memory cells acts as a single synapse having a weight value expressed as an output current Iout defined by the sum of the weight values stored in the memory cells for the row.

24 illustrates a configuration of an array of two-gate memory cells (ie, such as those shown in FIG. 1) configured as source sum matrix multipliers. The lines for the array of Figure 24 are the same as the one of Figure 5 except that the control gate 22a operates vertically and both of the control gates are used for each row of memory cells. Specifically, each row of memory cells includes two control gates: a first control gate 22a1 that connects all of the control gates 22a of the odd-numbered memory cells; and a second control gate 22a2 that has an even-numbered column All of the control gates 22a of the memory unit are connected together.

The matrix inputs for this configuration are Vin0...VinN and placed on control gates 22a1 and 22a2. Specifically, the input Vin0 is placed on the control gate 22a1 for the odd column cells in row 1. Vin1 is placed on control gate 22a2 for the even column elements in row 1. Vin2 is placed on the control gate 22a1 for the odd column elements in row 2. Vin3 is placed on control gate 22a2 for even column cells in row 2, and so on. The matrix outputs Iout0...IoutN are generated on the source line 14a. For each pair of memory cells sharing the shared source line 14a, the output current will be the differential input of the top cell minus the bottom cell. Out. Therefore, for this architecture, each pair of memory cells acts as a single synapse with a weight value, which is represented as an output current Iout, which is stored in a pair of memory cells. The sum of the differential outputs specified by the weight values in the memory cells.

Exemplary operating voltages for the embodiments of Figures 15-16, 19, and 20 include:

Approximate values include:

Exemplary operating voltages for the embodiments of Figures 17-18 and 22 include:

Approximate values include:

Figure 25 illustrates an exemplary current-voltage logarithmic converter 50 (WL = select gate line, CG = control gate line, EG = erase gate line) for use with the present invention. The memory is biased in the weak reversal zone, Ids = Io * e ^{(Vg - Vth) / kVt} . Figure 26 illustrates an exemplary voltage-current logarithmic converter 52 for use with the present invention. The memory is biased in the weak reversal zone. Figure 27 illustrates a current summer 54 of reference Gnd for use with the present invention. Figure 28 below illustrates a current summer 56 of reference Vdd for use with the present invention. Examples of loads include diodes, non-volatile memory cells, and resistors.

The above memory array configuration implements a feedforward classification engine. Training is accomplished by storing the "weight" value in a memory unit (generating a synaptic array), which means that the sub-threshold slope factor of the individual unit has been modified. Neurons are implemented by summing the outputs of the synapses and depending on the neuron threshold to emit or not to emit (ie, make a decision).

The following steps can be used to process the input current I _E (for example, the input current is directly from the output of the feature calculation for image recognition):

Step 1 - Convert to a logarithmic scale to make it easier to take advantage of non-volatile memory processing.

‧ Use bipolar transistor input current to voltage conversion. The bias voltage V _{BE of the} bipolar transistor has a logarithmic relationship with the emitter current.

‧VBE=a*lnI _E -b → V _BE lnI _E

- where a (ratio) and b (offset or offset) are constant

The _VBE voltage is generated such that the memory cells will operate in the sub-threshold region.

Step 2 - Apply the generated bias voltage VBE to the word line (in the sub-threshold region).

‧ CMOS transistor output current I _汲 has an exponential relationship with input voltage (V _GS ), thermal voltage (U _T ) and kappa (k = C _ox / (C _ox + C _dep )), where C _ox and C _dep Linearly related to the charge on the floating gate.

‧I _Drain Exp(kV _BE /U _T ), or

‧LnI _Drain kV _BE /U _T

‧ The logarithmic I _汲 has a linear relationship with multiple V _BEs and the charge on the floating gate (related to kappa), where U _T is constant at a given temperature.

‧ For synapses, there is an output = input * weight relationship.

The outputs of each of the cells (I- _poles ) can be tied together in a read mode to add the values of the synapses in the array or segment of the array. Once the I- _thole has been added, it can be fed into the current comparator and output "logic" 0 or 1 depending on the comparison to a single perceptual neural network. The above describes a perception (a segment). The output from each perception can be fed to a set of segments for multiple perceptions.

In a memory-based convolutional neural network, a set of inputs needs to be multiplied by certain weights to produce the desired result of the hidden layer or the output layer. As explained above, one technique is to scan the aforementioned image (e.g., an N x N matrix) using an M x M filter (core) that shifts X pixels across the image in horizontal and vertical directions. The scanning of the pixels can be performed at least partially simultaneously as long as there is sufficient input to the memory array. For example, as shown in FIG. 29, a filter size of M=6 (i.e., a 6 x 6 array of 36 pixels) can be used to scan an N x N image array using a shift of X=2. In the example, the first six inputs of the memory array to N ² inputs provide a first column of six pixels in the filter. Then, the filter is provided in the second column of six pixels to the N ² inputs of the second input of the N first input 6, and the like. This is shown in the first column of the graph in Figure 29, where the dots represent the weights stored in the memory array for multiplication of the inputs set forth above. Then, the filter is shifted to the right by two pixels, and the third to eighth inputs of the first N inputs are provided with a first column of six pixels in the shift filter, and a third of the second N inputs The eighth input provides the second column of six pixels, and so on. Once the filter has been shifted to the right of the image, the filter is reset back to the left, but shifted down by two pixels, where the process is repeated again until the entire N x N image is scanned. The scanning of each group of horizontal shifts can be represented by a trapezoidal shape that shows which of the N ² memory array inputs has data for multiplication.

Therefore, an N x N image array using a shift of two pixels between scans and a 6×6 filter size scan requires N ² inputs and ((N-4)/2)) ² columns. Figure 30 graphically illustrates a trapezoidal shape indicating how the weights in the memory array are stored for filter scanning. The shaded areas of each column represent the weights applied to the input during a set of horizontal scans. The arrows indicate linear input lines of the memory array (eg, input line 28a in Figure 15 that receives input data that extends in a linear fashion across the memory array, each of which always accesses the same memory cell column; In the case of an array, each of the input lines always access the same memory cell row). The white area indicates that no data is being supplied to the input. Therefore, the white area indicates an invalid use of the memory cell array.

By reassembling the memory array, as shown in Figure 31, efficiency can be increased and the total number of inputs reduced. In particular, the input lines of the memory array are periodically shifted to another column or row, thereby reducing the unused portion of the array and thus reducing the number of repeated input lines above the array where scanning is required. Specifically, in the case where the example of shifting X=2, the arrow indicates that each input line periodically moves through two columns or two rows, thereby converting the trapezoidal shape of the memory cell using a wide interval into closely spaced. The rectangular shape used by the memory unit. Although the harness requires additional space between the memory cell portions to implement this shift, the number of inputs required in the memory cell array is greatly reduced (only 5n+6).

Figure 32 illustrates the array of Figure 15, but with periodic shifts for the two columns of lines 28a used as input lines. The periodic shift for the column of input lines can be seen in Figure 17, Figure 22 and the array of Figure 23 are similarly implemented. Figure 33 illustrates the array of Figure 20, but with periodic shifts for the two rows of lines 28a1 and 28a2 used as input lines. The periodic shifts for the rows of input lines can be similarly implemented in the arrays of Figures 19, 21 and 24.

It is to be understood that the invention is not limited to the embodiment(s) described above and described herein, and encompasses any variant or all variants falling within the scope of any one of the claims. For example, the description of the present invention is not intended to limit the scope of the claims or the scope of the claims, but only to recite one or more technical features that may be covered by one or more claims. The above examples of materials, procedures and numerical examples are illustrative only and should not be considered as limiting the scope of the claims. A single material layer can be formed into a plurality of layers having such or similar materials, and vice versa. Although the output of each memory cell array is manipulated by filter condensation before being sent to the next neuron layer, they need not be.

It should be noted that, as used herein, the terms "over" and "on" are used to include "directly on" ( Directly on) and the meaning of "indirectly on" (with in-between materials, components or intervals). Similarly, the word "adjacent" includes "directly adjacent" (without the centering material, component or spacing disposed therebetween) and "indirectly adjacent" (with centered material, component or spacing) In the meantime, the term "mounted to" includes "directly mounted to" (without the centering of materials, components or spaces) and "indirectly installed" (indirectly Mounted to) (with the centered material, component or spacing set in between), and "electrically coupled (electrically coupled The term "coupled" includes "directly electrically coupled to" (in the case of a non-centered material or component in which the components are electrically connected) and "indirectly electrically coupled to" (There is the meaning of having a centered material or component that electrically connects the components therebetween). For example, forming an element "on top of a substrate" can include the formation of elements directly on the substrate without the presence of materials/components in between, and the indirect formation of elements on the substrate with one or more centered materials/components therebetween. presence.

31‧‧‧Digital-to-analog converter

32a~e‧‧‧Enter VMM

Claims (31)

A neural network device comprising: a first plurality of synapses configured to receive a first plurality of inputs, and generating a first plurality of outputs from the first plurality of inputs, wherein the first The plurality of synapses comprise: a plurality of memory cells, wherein each of the memory cells comprises a spaced apart source region and a drain region formed in a semiconductor substrate, and a channel region is in the source region Extending between the drain regions; a floating gate disposed above and insulated from the first portion of the channel region; and a non-floating gate disposed above the second portion of the channel region and The second portion is insulated; each of the plurality of memory cells configured to store a weight value corresponding to a plurality of electrons on the floating gate; the plurality of memory cells configured to cause the first plurality The inputs are multiplied by the stored weight values to produce the first plurality of outputs; a first plurality of neurons configured to receive the first plurality of outputs. The neural network device of claim 1, wherein the first plurality of neurons are configured to generate a first plurality of decisions based on the first plurality of outputs. The neural network device of claim 2, further comprising: a second plurality of synapses configured to receive a second plurality of inputs based on the first plurality of decisions, and from the second plurality of inputs Generating a second plurality of outputs, wherein the second plurality of synapses comprises: a plurality of second memory cells, wherein each of the second memory cells includes a second source region and a second drain region formed in the semiconductor substrate, and a second channel region Extending between the second source region and the second drain region; a second floating gate disposed above the first portion of the second channel region and insulated from the first portion; and a second non-floating gate Provided above and insulated from the second portion of the second channel region; each of the plurality of second memory cells configured to store a plurality of electrons corresponding to the second floating gate a second weight value; the plurality of second memory units configured to multiply the second plurality of inputs by the stored second weight values to generate the second plurality of outputs; A plurality of neurons configured to receive the second plurality of outputs. The neural network device of claim 3, wherein the second plurality of neurons are configured to generate a second plurality of decisions based on the second plurality of outputs. The neural network device of claim 1, wherein each of the memory cells of the first plurality of synapses further comprises: a second non-floating gate disposed above the source region and the source Zone insulation; and a third non-floating gate disposed above the floating gate and insulated from the floating gate. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates in one of the columns of the memory cells; a plurality of second lines each electrically connecting the second non-floating gates of one of the rows of the memory cells; a plurality of third lines each of the memory cells The third non-floating gates of one of the columns are electrically connected together; a plurality of fourth lines each of the source regions of one of the columns of the memory cells Electrically connected together; a plurality of fifth lines each electrically connecting the one of the rows of the memory cells; wherein the first plurality of synapses are configured The first plurality of inputs are received on the plurality of first lines, and the first plurality of outputs are provided on the plurality of fifth lines. The neural network device of claim 6, wherein for each of the plurality of fifth lines, one of the first plurality of outputs is provided thereon for all of the memory cells in a row In the memory unit, the one of the first plurality of outputs is a sum of currents passing through the memory cells multiplied by respective weight values stored in the memory cells. The neural network device of claim 6, further comprising: circuitry for converting the current logarithm of the first plurality of inputs to the first plurality of inputs before the receiving the first plurality of inputs on the plurality of first lines Voltage. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates in one of the columns of the memory cells; a plurality of second lines each electrically connecting the second non-floating gates of one of the columns of the memory cells; a plurality of third lines each of the memory cells The third non-floating gates of one of the columns are electrically connected together; a plurality of fourth lines each of the source regions of one of the rows of the memory cells Electrically connected together; a plurality of fifth lines each electrically connecting the one of the rows of the memory cells; wherein the first plurality of synapses are configured The first plurality of inputs are received on the plurality of first lines, and the first plurality of outputs are provided on the plurality of fifth lines. The neural network device of claim 9, wherein for each of the plurality of fifth lines, one of the second plurality of outputs is provided thereon for all of the memory cells in a row In the memory unit, the one of the second plurality of outputs is a sum of currents passing through the memory cells multiplied by respective weight values stored in the memory cells. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates of the odd-numbered column memory cells in one of the rows of the memory cells; the plurality of second wires, each of the memory cells The first non-floating gates of the even-numbered column memory cells of one of the rows are electrically connected together; a plurality of third lines each electrically connecting the second non-floating gates of one of the rows of the memory cells; a plurality of fourth lines each of the memory cells The third non-floating gates of one of the columns are electrically connected together; a plurality of fifth lines each of the source regions of one of the columns of the memory cells Electrically connected together; a plurality of sixth lines each electrically connecting the one of the ones of the rows of the memory cells; wherein the first plurality of synapses are configured Receiving some of the first plurality of inputs on the plurality of first lines, and receiving the other of the first plurality of inputs on the plurality of second lines, and providing the plurality of fifth lines The first plurality of outputs. The neural network device of claim 11, wherein for each of the plurality of fifth lines, one of the first plurality of outputs is provided thereon for all of the columns of the memory cells In the memory unit, the one of the first plurality of outputs is a sum of currents passing through the memory cells multiplied by respective weight values stored in the memory cells. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates of the odd-numbered column memory cells of one of the rows of the memory cells; a plurality of second lines each electrically connecting the first non-floating gates of the even-numbered column memory cells of one of the rows of the memory cells; a plurality of third lines, each of which The second non-floating gates of one of the rows of the memory cells are electrically connected together; a plurality of fourth wires each of the ones of the memory cells The third non-floating gates are electrically connected together; a plurality of fifth lines each electrically connecting the source regions of one of the columns of the memory cells together; a six-wire, each of which electrically connects the odd-numbered drain regions of one of the rows of the memory cells; a plurality of seventh lines, each of the rows of the memory cells An even number of drain regions in one are electrically coupled together; wherein the first plurality of synapses are configured to receive some of the first plurality of inputs on the plurality of first lines, and in the plurality of Receiving the other of the first plurality of inputs on the second line, and providing the plurality of fifth lines A plurality of outputs. The neural network device of claim 13, wherein for each of the plurality of fifth lines, one of the first plurality of outputs is provided thereon for all of the columns of the memory cells The memory cell pair, the one of the first plurality of outputs being a sum of differential outputs from the pair of memory cells, and wherein each of the differential outputs is a pair passing through the memory cell The current is multiplied by a difference between the respective weight values stored in the pair of memory cells. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates of the odd-numbered column memory cells in one of the rows of the memory cells; the plurality of second wires, each of the memory cells The first non-floating gates of the even-numbered column memory cells of one of the rows are electrically connected together; the plurality of third wires, each of the ones of the columns of the memory cells Waiting for the second non-floating gates to be electrically connected together; a plurality of fourth lines each of the third non-floating gates of the odd-numbered column memory cells of one of the rows of the memory cells a plurality of fifth lines each electrically connecting the third non-floating gates of the even-numbered column memory cells of one of the rows of the memory cells; the plurality of sixth lines Electrically connecting the source regions of one of the columns of the memory cells Connected together; a plurality of seventh lines each electrically connecting the odd-numbered drain regions of one of the rows of the memory cells; a plurality of eighth lines each of the memory An even bungee region of one of the rows of cells is electrically coupled together; wherein the first plurality of synapses are configured to receive the first plurality of inputs on the plurality of first lines And receiving the other of the first plurality of inputs on the plurality of second lines, and providing the first plurality of outputs on the plurality of sixth lines. The neural network device of claim 15, wherein for each of the plurality of sixth lines, one of the first plurality of outputs is provided thereon for all of the columns of the memory cells The memory cell pair, the one of the first plurality of outputs being a sum of differential outputs from the pair of memory cells, and wherein each of the differential outputs is a pair passing through the memory cell The current is multiplied by a difference between the respective weight values stored in the pair of memory cells. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates of one of the columns of the memory cells together; a plurality of second wires each of the ones of the memory cells The second non-floating gates are electrically connected together; a plurality of third lines each electrically connecting the third non-floating gates in one of the columns of the memory cells; a fourth line electrically connecting the source regions of one of the rows of the memory cells together; a plurality of fifth lines, each of the memory cells The bungee regions of one of the rows are electrically connected together; wherein the first plurality of synapses are configured to receive the first plurality of inputs on the plurality of first lines, and at the plurality of The first plurality of outputs are provided on the four lines. The neural network device of claim 17, wherein for each of the plurality of fourth lines, one of the first plurality of outputs is provided thereon, for a row of the memories For all of the memory cells in the volume unit, the one of the first plurality of outputs is the current through the memory cells multiplied by the respective weight values stored in the memory cells One and one. The neural network device of claim 1, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates of one of the columns of the memory cells together; a plurality of second wires each of the ones of the memory cells The source regions are electrically connected together; a plurality of third lines each electrically connecting the drain regions of one of the rows of the memory cells; wherein the first plurality The synapse is configured to receive the first plurality of inputs on the plurality of first lines and to provide the first plurality of outputs on the plurality of third lines. The neural network device of claim 19, wherein for each of the plurality of third lines, one of the first plurality of outputs is provided thereon for all of the memory cells in a row In the case of a memory unit, the one of the first plurality of outputs is a sum of currents passing through the memory cells multiplied by respective weight values stored in the memory cells. The neural network device of claim 1, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Electrically connecting the first non-floating gates of the odd-numbered column memory cells of one of the rows of the memory cells; a plurality of second lines each electrically connecting the first non-floating gates of the even-numbered column memory cells of one of the rows of the memory cells; a plurality of third lines, each of which The source regions of one of the columns of the memory cells are electrically connected together; a plurality of fourth wires, each of the ones of the rows of the memory cells The first bungee regions are electrically connected together; wherein the first plurality of synapses are configured to receive some of the first plurality of inputs on the plurality of first lines and receive the plurality of second lines on the plurality of first lines The other of the first plurality of inputs, and the first plurality of outputs are provided on the plurality of third lines. The neural network device of claim 21, wherein for each of the plurality of third lines, one of the first plurality of outputs is provided thereon for all of the memories in a column of the memory cells The body unit pair, the one of the first plurality of outputs being a sum of differential outputs from the pair of memory cells, and wherein each of the differential outputs is through the memory unit pair The current is multiplied by a difference between the respective weight values stored in the pair of memory cells. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Some but not all of the first non-floating gates of one of the columns of the memory cells, and some but not all of the other of the columns of the memory cells The first non-floating gates are electrically connected together; a plurality of second lines each electrically connecting the second non-floating gates of one of the rows of the memory cells; a plurality of third lines each of the memory cells The third non-floating gates of one of the columns are electrically connected together; a plurality of fourth lines each of the source regions of one of the columns of the memory cells Electrically connected together; a plurality of fifth lines each electrically connecting the one of the rows of the memory cells; wherein the first plurality of synapses are configured The first plurality of inputs are received on the plurality of first lines, and the first plurality of outputs are provided on the plurality of fifth lines. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Some but not all of the first non-floating gates of one of the columns of the memory cells, and some but not all of the other of the columns of the memory cells The first non-floating gates are electrically connected together; a plurality of second wires each electrically connecting the second non-floating gates in one of the columns of the memory cells; a third line electrically connecting the third non-floating gates of one of the columns of the memory cells together; a plurality of fourth lines each of the memory cells The source regions of one of the rows are electrically connected together; a plurality of fifth lines each electrically connecting the one of the ones of the rows of memory cells together; wherein the first plurality of synapses are configured to be in the plurality of The first plurality of inputs are received on the first line, and the first plurality of outputs are provided on the plurality of fifth lines. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Some but not all of the first non-floating gates of the odd-numbered memory cells of one of the rows of the memory cells, and the other of the rows of the memory cells Some, but not all, of the first non-floating gates of the odd-numbered memory cells are electrically connected together; a plurality of second lines, each of which is an even number of one of the rows of the memory cells Some but not all of the first non-floating gates of the column memory cells, and some but not all of the even-numbered column memory cells of the other of the rows of the memory cells Non-floating gates are electrically connected together; a plurality of third wires each electrically connecting the second non-floating gates of one of the rows of the memory cells; a plurality of fourth wires, The third non-floating electrical connection in one of the columns of the memory cells Connected together; a plurality of fifth lines each electrically connecting the source regions of one of the columns of the memory cells; a plurality of sixth lines, each of the memories The bungee regions of one of the rows of body units are electrically connected together; Wherein the first plurality of synapses are configured to receive some of the first plurality of inputs on the plurality of first lines, and receive the other of the first plurality of inputs on the plurality of second lines And providing the first plurality of outputs on the plurality of fifth lines. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Some but not all of the first non-floating gates of the odd-numbered memory cells of one of the rows of the memory cells, and the other of the rows of the memory cells Some, but not all, of the first non-floating gates of the odd-numbered memory cells are electrically connected together; a plurality of second lines, each of which is an even number of one of the rows of the memory cells Some but not all of the first non-floating gates of the column memory cells, and some but not all of the even-numbered column memory cells of the other of the rows of the memory cells Non-floating gates are electrically connected together; a plurality of third wires each electrically connecting the second non-floating gates of one of the rows of the memory cells; a plurality of fourth wires, The third non-floating electrical connection in one of the columns of the memory cells Connected together; a plurality of fifth lines each electrically connecting the source regions of one of the columns of the memory cells; a plurality of sixth lines, each of the memories The odd bungee regions of one of the rows of body units are electrically connected together; a plurality of seventh lines each electrically connecting together even-numbered drain regions of one of the rows of memory cells; wherein the first plurality of synapses are configured to be in the plurality of One of the first plurality of inputs is received on a line, and the other of the first plurality of inputs is received on the plurality of second lines, and the first plurality of outputs are provided on the plurality of fifth lines. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Some but not all of the first non-floating gates of the odd-numbered memory cells of one of the rows of the memory cells, and the other of the rows of the memory cells Some, but not all, of the first non-floating gates of the odd-numbered memory cells are electrically connected together; a plurality of second lines, each of which is an even number of one of the rows of the memory cells Some but not all of the first non-floating gates of the column memory cells, and some but not all of the even-numbered column memory cells of the other of the rows of the memory cells The non-floating gates are electrically connected together; a plurality of third wires each electrically connecting the second non-floating gates of one of the columns of the memory cells; a plurality of fourth wires, The odd column memory cells of one of the rows of the memory cells And the third non-floating gates are electrically connected together; the plurality of fifth lines each of the third non-floating gates of the even-numbered memory cells of one of the rows of the memory cells Together a plurality of sixth lines each electrically connecting the source regions of one of the columns of the memory cells; a plurality of seventh lines each of the memory cells An odd-numbered drain region of one of the rows is electrically connected together; a plurality of eighth wires each electrically connecting the even-numbered drain regions of one of the rows of the memory cells; Wherein the first plurality of synapses are configured to receive some of the first plurality of inputs on the plurality of first lines, and receive the other of the first plurality of inputs on the plurality of second lines And providing the first plurality of outputs on the plurality of sixth lines. The neural network device of claim 5, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Some but not all of the first non-floating gates of one of the columns of the memory cells, and some but not all of the other of the columns of the memory cells The first non-floating gates are electrically connected together; a plurality of second wires each electrically connecting the second non-floating gates in one of the columns of the memory cells; a third line electrically connecting the third non-floating gates of one of the columns of the memory cells together; a plurality of fourth lines each of the memory cells The source regions of one of the rows are electrically connected together; a plurality of fifth lines each electrically connecting the one of the ones of the rows of memory cells together; wherein the first plurality of synapses are configured to be in the plurality of The first plurality of inputs are received on the first line, and the first plurality of outputs are provided on the plurality of fourth lines. The neural network device of claim 1, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of which Some but not all of the first non-floating gates of one of the columns of the memory cells, and some but not all of the other of the columns of the memory cells The first non-floating gates are electrically connected together; a plurality of second lines each electrically connecting the source regions of one of the rows of the memory cells together; a three-wire, each of the one of the rows of the memory cells being electrically connected together; wherein the first plurality of synapses are configured to receive on the plurality of first lines The first plurality of inputs, and the first plurality of outputs are provided on the plurality of third lines. The neural network device of claim 19, wherein for each of the plurality of third lines, one of the first plurality of outputs is provided thereon for all of the memory cells in a row In the case of a memory unit, the one of the first plurality of outputs is a sum of currents passing through the memory cells multiplied by respective weight values stored in the memory cells. The neural network device of claim 1, wherein the memory cells of the first plurality of synapses are configured as columns and rows, and wherein the first plurality of synapses comprise: a plurality of first lines, each of the odd-numbered memory cells of one of the rows of the memory cells, and not all of the first non-floating gates, and the memory cells Some of the odd-numbered memory cells of the other of the rows, but not all of the first non-floating gates, are electrically connected together; a plurality of second lines, each of the memory cells Some of the even-numbered column memory cells in one of the rows, but not all of the first non-floating gates, and some of the even-numbered column memory cells in the other of the rows of the memory cells And not all of the first non-floating gates are electrically connected together; a plurality of third lines each electrically connecting the source regions of one of the columns of the memory cells together; a fourth line electrically connecting the one of the ones of the rows of the memory cells together; wherein the first plurality of synapses are configured to be in the plurality of Receiving some of the first plurality of inputs on a line and receiving on the plurality of second lines A first plurality of inputs of the other person, and providing the first plurality of outputs of the plurality of third lines.