CN119207518B

CN119207518B - Method, device and storage system for performing in-memory computing

Info

Publication number: CN119207518B
Application number: CN202410001792.8A
Authority: CN
Inventors: 孙楠楠
Original assignee: Baidai Shanghai Data Technology Co ltd
Current assignee: Baidai Shanghai Data Technology Co ltd
Priority date: 2024-01-02
Filing date: 2024-01-02
Publication date: 2025-06-20
Anticipated expiration: 2044-01-02
Also published as: CN119207518A

Abstract

The present invention relates to a method, device and storage system for performing in-memory calculations. A storage transistor array is used as a carrier of a weight matrix of a neural network in a flash memory for performing in-memory calculations. Input data of the neural network is set to be transmitted to a series of bit lines of the storage transistor array, and output data of the neural network is set to be captured from a series of source lines of the storage transistor array. The weight of any transistor in the storage transistor array is adjusted by its threshold voltage, thereby dynamically configuring the weight matrix of the neural network.

Description

Method, device and storage system for executing in-memory calculation

Technical Field

The invention mainly relates to the technical field of data storage, in particular to a method for performing in-memory calculation in the field of data storage, an in-memory calculation device and a corresponding storage system.

Background

In the prior electronic device, the calculation and the storage are separated, and the calculation part such as a processor and the storage part such as a hard disk or a memory and the like are respectively focused on the task, so that the calculation and the storage do not interfere with each other basically. There are also manufacturers that integrate processors and memories in close physical locations for increasing computing speed and computing power, for example, in circuit board layout designs where they are close together, short circuits, and are stacked directly on top of each other during the packaging stage.

The power improvement in the industry has been mainly advanced gradually according to the rhythm of moore's law, but the mainstream calculation paradigm always follows the von neumann architecture. As a new computing power, the industry mainly focuses on the narrow computing integration, that is, computing by using a storage medium, but the computing medium method is not completely unified. The integrated calculation is used as novel calculation power, and is hopeful to solve the problems of compiling walls, storage walls and power consumption walls under the traditional von neumann architecture. There is a consensus in the industry that it has basically been determined as one of the key technologies to solve the computational problem.

The storage and calculation are organically integrated, and the method is expected to become an advanced calculation architecture scheme in the computing age based on the huge energy efficiency ratio and the promotion potential. There is a doubt as to how to map the memory to the neural network and how to configure the neural network with a dynamically adjustable weight matrix to adapt to the actual application of the neural network, such as back propagation, etc., which is a key issue that must be addressed with in-memory computation. In particular, the problem of solving the weight offset and the like which are faced when the weight of the internal neural network of the memory is updated is solved urgently, and the problem is hidden for the memory.

Disclosure of Invention

The application relates to a method for performing in-memory calculations, comprising:

In the flash memory, the memory transistor array is used as the carrier of the weight matrix of the neural network, a series of bit lines for transmitting the input data of the neural network to the memory transistor array and a series of output data which is extracted from the memory transistor array to the neural network are set, wherein the weight of any transistor in the memory transistor array is regulated by the threshold voltage of the transistor array, so that the weight matrix of the neural network is dynamically configured.

The method comprises the steps that the storage transistor array used for the neural network is located on the same physical block of the flash memory, and the storage space where input data and output data of the neural network are respectively located and the storage space where the storage transistor array is located respectively belong to different physical blocks of the flash memory.

In the above method, the original input data of the neural network is converted into input quantities in the form of voltage values to be supplied to a series of bit lines of the memory transistor array based on the input quantities ordered into vectors.

In the method, the output data of the neural network is characterized by the output currents of the memory transistor array, and the output currents of a series of source lines of the memory transistor array are regarded as output quantities of the memory transistor array which are sequenced into vectors.

The method includes changing the floating gate injection charge, and adjusting the weight of any transistor at the location in the memory transistor array (i.e. at the location of an element corresponding to any transistor in the weight matrix) by changing the floating gate injection charge of the transistor.

In the method, in the Back propagation (Back propagation) stage of the neural network, the weight matrix of the neural network is dynamically configured by resetting the weight sizes at various positions in the memory transistor array.

In the above method, a digital-to-analog converter is configured at each bit line of the memory transistor array, and is used for converting input data of the neural network into a voltage value matched with an input signal pattern of the bit line.

In the above method, an analog-to-digital converter is configured at each source line of the storage transistor array, and is used for converting an output signal in an output current format at the source line into output data adapted to a digital output quantity of the neural network.

The application relates to a device for executing in-memory computation, which is characterized by comprising:

a flash memory, wherein the memory transistor array is used as a carrier of a weight matrix of the neural network, a series of bit lines for transmitting input data of the neural network to the memory transistor array are set, and output data of the neural network is set and retrieved from a series of source lines of the memory transistor array;

An input unit converting original input data of the neural network into input quantities in the form of voltage values to be supplied to a series of bit lines of the memory transistor array based on the input quantities ordered into vectors;

An output unit for converting an output signal in an output current format at the source lines into output data adapted to the digital output quantity of the neural network, the output currents of a series of source lines of the memory transistor array being regarded as a vectorized output quantity of the memory transistor array.

The device for executing in-memory calculation, wherein the storage transistor array for the neural network is positioned on the same physical block of the flash memory, and the storage space where the input data and the output data of the neural network are respectively positioned and the storage space where the storage transistor array is positioned respectively belong to different physical blocks of the flash memory.

The above-described apparatus for performing in-memory calculations, wherein raw input data of the neural network is converted into input quantities in the form of voltage values to be supplied to a series of bit lines of the memory transistor array based on the input quantities ordered into vectors.

The apparatus for performing in-memory calculations described above, wherein the output data of the neural network is characterized by an output current of the memory transistor array, the output currents of a series of source lines of the memory transistor array (transistor arrays) are considered as a vectorized output of the memory transistor array.

The above-mentioned apparatus for performing in-memory computation, wherein the means for adjusting the weight of any transistor includes changing the floating gate injection charge, and adjusting the weight of the any transistor at the location in the memory transistor array (i.e. at the location of an element corresponding to the any transistor in the weight matrix) by changing the floating gate injection charge of the transistor.

The above-described apparatus for performing in-memory computation may dynamically configure a weight matrix of the neural network by resetting the weight size at each location in the memory transistor array in a Back propagation (Back propagation) stage of the neural network.

In the above-mentioned apparatus for performing in-memory computation, the input unit includes at least a digital-to-analog converter, and each bit line of the memory transistor array is configured with a digital-to-analog converter for converting input data of the neural network into a voltage value matched with an input signal pattern of the bit line.

In the above-mentioned device for performing in-memory computation, the output unit at least includes an analog-to-digital converter, and each source line of the storage transistor array is configured with an analog-to-digital converter, which is used for converting an output signal in an output current format at the source line into output data adapted to a digital output quantity of the neural network.

The application relates to a storage system for performing in-memory calculations, comprising:

A memory chip based on a flash memory, wherein the memory transistor array is used as a carrier of a weight matrix of the neural network in the flash memory, wherein the weight of any transistor is adjusted by using the threshold voltage of the memory transistor array, so that the weight matrix of the neural network is dynamically configured;

A digital-to-analog converter built in the memory chip, wherein the digital-to-analog converter is configured at each bit line of the memory transistor array to convert input data of the neural network into a voltage value matched with an input signal pattern of the bit line;

An analog-to-digital converter built in the memory chip, wherein the analog-to-digital converter is configured at each source line of the memory transistor array to convert an output signal in an output current format at the source line into output data adapted to a digital output quantity of the neural network.

The memory system comprises a memory transistor array used for the neural network, a memory system and a memory system, wherein the memory transistor array used for the neural network is positioned on the same physical block of the flash memory, and the memory space where input data and output data of the neural network are respectively positioned and the memory space where the memory transistor array is positioned respectively belong to different physical blocks of the flash memory.

In the above memory system, the original input data of the neural network is converted into input quantities in the form of voltage values, and the input quantities are supplied to a series of bit lines of the memory transistor array based on the ordered vector.

In the above memory system, the output data of the neural network is characterized by the output currents of the memory transistor array, and the output currents of the series of source lines of the memory transistor array are regarded as the output quantities of the memory transistor array which are sequenced into vectors.

In the above memory system, the method for adjusting the weight of any transistor includes changing the floating gate injection charge, and adjusting the weight of any transistor at the location in the memory transistor array (i.e. at the location of an element corresponding to any transistor in the weight matrix) by changing the floating gate injection charge of the transistor.

In the above memory system, the weight matrix of the neural network is dynamically configured by resetting the weight size at each position in the memory transistor array in the Back propagation (Back propagation) stage of the neural network.

In the above memory system, a digital-to-analog converter is configured at each bit line of the memory transistor array, and is used for converting input data of the neural network into a voltage value matched with an input signal pattern of the bit line.

In the above memory system, each source line of the memory transistor array is configured with an analog-to-digital converter, which is used for converting an output signal in an output current format at the source line into output data adapted to a digital output quantity of the neural network.

The application relates to a chip for executing in-memory computation, which is characterized by comprising:

the system comprises a memory transistor array, a set of input data of the neural network, a set of bit lines of the memory transistor array, a set of output data of the neural network, a set of data storage and a set of data storage, wherein the memory transistor array is used as a carrier of a weight matrix of the neural network, the weight of any transistor is adjusted by using threshold voltage of the memory transistor array, so that the weight matrix of the neural network is dynamically configured;

A digital-to-analog converter integrated in the chip, wherein the digital-to-analog converter is configured at each bit line of the memory transistor array to convert input data of the neural network into a voltage value matched with an input signal pattern of the bit line;

An analog-to-digital converter integrated in the chip, wherein an analog-to-digital converter is configured at each source line of the array of storage transistors to convert an output signal in an output current format at the source line into output data adapted to the digital output of the neural network.

Based on the foregoing, one of the greatest advantages of the present application is to map the weight matrix of the neural network directly to the memory transistor array and by adjusting the transistor threshold voltages, the weight matrix of the neural network can be dynamically configured to adapt to weight matrix (weight matrix) updates of the neural network such as matching back propagation. And thus, the difficult problems of weight offset and the like faced when the weight of the internal neural network of the memory is updated can be solved, and one or more elements in the weight matrix mapped by the memory are prevented from generating unexpected hidden offset.

Based on the foregoing, the second most advantage of the present application is that the compiling wall, the storage wall and the power consumption wall under the traditional von neumann architecture can be solved by adopting the non-von neumann architecture. One convenient point in compiling is to realize programming of core weights by adjusting threshold voltages of transistors, and the solution can realize core weight class compiling only by configuring voltages or currents of three ends of the transistors, and has no obstacle in designing a part of compiling behaviors because compiling compatible with information quantities such as voltages and currents is carried out on a traditional memory. Furthermore, unlike traditional scheme, the present application needs no memory and memory wall to execute data exchange between the memory and the processor, and the present application has data storage and data calculation, especially matrix operation, inside the memory and thus has no memory wall problem. The specific implementation of the neural network is based on the premise that a large amount of data carrying is not required to be implemented between a memory and a processor, and the high energy consumption problem of the neural network and the artificial intelligence application scene thereof is also solved in view of the fact that most of energy consumption of a power consumption wall generates, for example, data carrying power consumption which is thousands times or more of calculation power consumption in the data carrying process. In addition, the AI operation needs 1PB/s, and the data carrying speed, such as DRAM 40GB-1TB/s, is far insufficient to meet the AI operation requirement, if the memory internal calculation as the application is used, the problem that the data carrying speed cannot follow the AI operation speed can be solved better.

Based on the foregoing, the third most advantage of the present application is that the propulsive effort is greatly improved. By using the integrated technology of memory calculation, a large amount of weight parts with AI calculation and multiplication and addition calculation can be stored in a memory unit, and the existing memory circuit part of the memory unit is modified, so that data input and calculation processing are performed at the same time of reading, and a neural network such as convolution operation and the like is completed in a memory array. Since the convolution operation of a large number of multiply-add is a core component of deep learning, in-memory computation and in-memory logic are well suited for artificial intelligence deep neural network applications and AI-based big data techniques.

Drawings

So that the manner in which the above recited objects, features and advantages of the present application can be understood in detail, a more particular description of the application, briefly summarized above, may be had by reference to the appended drawings.

Fig. 1 shows a part of transistors selected as a weight carrier of a neural network in a flash memory.

FIG. 2 is input data from a series of bit lines of a memory transistor array to a neural network.

Fig. 3 illustrates the input of pixel values or gray values to the memory transistor array, for example, as pictures or video.

Fig. 4 is a diagram of mapping a weight matrix of a neural network to a memory transistor array of a flash memory.

Fig. 5 is a graph of the weight values of the transistors in the transistor array adjusted by the change in the floating gate injection charge.

Fig. 6 is a diagram for coping with the weight shift and the like which are encountered when the weight of the internal neural network of the memory is updated.

Detailed Description

The technical solutions disclosed in the present application will be clearly and completely described in connection with the specific embodiments, but the described embodiments are only examples used for describing the present application and not all embodiments, and on the basis of these embodiments, those skilled in the art will recognize that any solution obtained without making an inventive effort is within the scope of protection of the present application.

Referring to fig. 1, a circuit of a general NOR flash memory device is known, which operates on the principle that a Floating Gate (Floating Gate) storing charges such as electrons is formed between a source S and a drain D of a memory transistor, that is, on a semiconductor in which current is unidirectionally conducted, and the Floating Gate is covered with a silicon oxide film insulator. Above the floating Gate is a select/Control Gate (Control Gate) that controls the conduction current between the source and drain. The charge of the floating gate, such as electrons, does not disappear due to power failure, so the flash memory is a nonvolatile memory. A write operation, such as applying a positive voltage to the control electrode, causes charge, such as electrons, to enter the floating gate through the insulating layer. The erase operation is reversed, such as applying a positive voltage to the substrate to pull electrons out of the floating gate to effect erasure. The data operation of the flash memory belongs to the prior art, and the data operation is not repeated herein, and the operations of reading, writing, erasing, and the like of the flash memory in the prior art can be applied to the flash memory of the application.

Referring to fig. 1, the flash memory is described in an array form in which the gates of each row of transistors are connected to a common word line and the drains of each column of transistors are connected to a common bit line and the sources of each column of transistors are connected to a common source line.

Referring to fig. 1, regarding the row word line example, the respective gates of the first row transistors are connected to word line G1 and the respective gates of the second row transistors are connected to word line G2, and the respective gates of the mth row transistors are connected to word line GM. Further omitted rows are not repeatedly shown in the figures. The total M rows of transistors are considered as row elements of the matrix mxn.

Referring to fig. 1, regarding the row source line example, the respective sources of the first row transistors are connected to the source line S1 and the respective sources of the second row transistors are connected to the source line S2, and the respective sources of the mth row transistors are connected to the source line SM. Further omitted rows are not repeatedly shown in the figures.

Referring to fig. 1, with respect to the column bit line example, the drains of the first column transistors are connected to bit line B1 and the drains of the second column transistors are connected to bit line B2 and the drains of the nth column transistors are connected to bit line BN. Further omitted columns are not repeated in the figures. The total N columns of transistors are considered as column elements of the matrix mxn.

Referring to fig. 1, the flash memory illustrated originally has the function of storing a memory symbol 1 or 0, in other words, the flash memory is not particularly useful in a conventional computer or server or other similar electronic device. The present application no longer regards flash memory as a memory space that only holds symbol 1 or 0, but instead claims that the circuit architecture of flash memory is changed to a class computing unit that can participate in artificial intelligence, with the underlying physical structure of flash memory being nearly unchanged.

Referring to fig. 2, the class computing unit based on the flash memory is used to replace the conventional computer, because the computer has several drawbacks in executing the operation stage of the neural network, namely, the data is read from and written to the memory outside the processor, the data handling time is hundreds to thousands times of the operation time, the reactive power consumption of the whole process is about 60% -90%, and the problem that the energy efficiency is very low and the memory wall is taken along becomes a big obstacle to the application of data computing. In particular, the greatest challenge in deep learning acceleration in neural networks is the frequent movement of data between the computing unit and the storage unit. Although parallel acceleration techniques, such as multi-core CPUs or many-core GPUs, are used to boost computing power, the latter molar age, memory bandwidth constrains the effective bandwidth of computing systems and results in a growing pace of overall computing power. The weight calculation of the neural network or the reasoning calculation of the neural network is operated on the traditional computer, so that time consumption and power consumption are one of the most prominent manifestations of the defects.

Referring to fig. 2, the neural network introduces a number of parameters that make its development bottleneck. The full connection layer connects each output neuron with all input neurons. It is assumed that if the number of input neurons is N and the number of output neurons is M, there are at least about N x M weights or M biases to train and calculate. The convolution layer is usually connected with the input neurons in the coverage range of the convolution kernel for each output neuron, and the convolution kernels used by all the outputs are consistent, so that the local connection and weight sharing reduce the parameters of the convolution layer, but a large amount of computation still needs to be performed under the condition of large-size image input and deep neural network. The reading and the calculation of a large amount of data face the memory bottleneck and the operation bottleneck of the network, the power consumption and the time delay of the neural network are increased sharply, and the application of the neural network is limited.

Referring to fig. 2, to address the aforementioned bottlenecks, the non-von neumann architecture illustrated may provide greater power and energy efficiency in a particular area beyond existing ASIC power chips. The memory calculation and the memory logic, namely the memory calculation integrated technology directly utilizes the memory to process or calculate the data, so that the data storage and the data calculation are fused in the same chip or the same area of the same memory, the bottleneck of the von neumann calculation architecture can be thoroughly eliminated, and the method is particularly suitable for the application scene of deep learning neural network, which is large-scale parallel with large data quantity.

Referring to FIG. 2, in the design with bit lines B1-BN as inputs and source lines S1-SM as outputs, the flash memory of FIG. 1 is considered as a physical carrier that can implement the neural network NET.

Referring to fig. 2, to illustrate a specific implementation, a matrix ma_wei is first designed as follows.

Referring to fig. 2, the bit line B1-BN inputs are represented by vectors ma_in= [ B1, B2. For convenience in mathematical expression, the input amount of bit line B1 is mapped to X ₁, the input amount of bit line B2 is mapped to X ₂, the input amount of bit line B3 is mapped to X ₃, and the input amount of bit line BN is mapped to X _N.

Referring to fig. 2, the source line S1-SM output is represented by vector ma_out= [ S1, S2..sm ]. For mathematical convenience, the output of source line S1 is mapped to Y ₁, the output of source line S2 is mapped to Y ₂, the output of source line S3 is mapped to Y ₃, and the output of source line SM is mapped to Y _M.

Referring to fig. 2, the purpose of the flash memory is to perform the operation function of the neural network NET, and the specific operation process is expressed by the following equation. Essentially this process is also a detailed example of simulating flash memory as a computing unit or class of computing units. This spirit is illustrated by Y ₁ and Y ₂ and Y _M, etc. as representative examples.

Y₁＝X₁*W₁₁+X₂*W₁₂+X₃*W₁₃+…X_N*W_1N

Y₂＝X₁*W₂₁+X₂*W₂₂+X₃*W₂₃+…X_N*W_2N

Y_M＝X₁*W_M1+X₂*W_M2+X₃*W_M3+…X_N*W_MN

Referring to fig. 2, a more intuitive expression is to use vector matrix multiplication to demonstrate the aforementioned complex operations. The vector matrix expressing flash memory performs local data processing of the neural network NET as follows.

Referring to fig. 2, if X ₁ to X _N are defined as input layers of the neural network NET, then Y ₁ to Y _N correspond to output layers of the neural network NET. It can be considered that the flash memory of fig. 1 has completed the whole vector operation from the input layer to the output layer at one time, and this operation is quite different from that of the conventional von neumann computer, because the operation only needs to measure the respective signal quantities from the source lines S1-SM, the single multiplication and addition do not need to be implemented one by one, and the final operation result can be fully reflected by the signal quantities from the source lines S1-SM at one time. So it is called a class calculation. The operation of a conventional von neumann computer then has to implement all multiplications and additions one by one, which is the greatest difference between them.

Referring to fig. 2, the neural network NET based on the flash memory is suitable for the field of image processing. It is assumed that data of pictures or videos (artificial intelligence, autopilot, face recognition, object detection) or the like with pixel values or gray values are input to a neural network NET based on a flash memory, hereinafter, an image gr_in is taken as an example.

Referring to fig. 3, the pixel points in the image gr_in are divided into a plurality of rows and a plurality of columns, a pixel corresponding to an address defining the 0 th pixel is g1, a pixel corresponding to an address defining the 1 st pixel is g2, a pixel corresponding to an address defining the 2 nd pixel is g3, and so on, a pixel corresponding to an end pixel address is gn. According to the arrangement rule and address rule of the pixel matrix, the gray value of any pixel point can be obtained by shooting the obtained image. The gray values or pixel values of the n pixels (belonging to the image gr_in) are allowed to be coupled to the flash memories X ₁ to X _N, etc., although the gray values or pixel values may be normalized or subjected to similar image data processing.

Referring to FIG. 3, one problem that is of concern is that the input amount format of the bit lines B1-BN should conform to the inherent characteristics of flash memory for flash memory, i.e., the input values of the bit lines B1-BN are not directly equivalent to the pixel values g 1-gn. In an alternative embodiment, the pixel values g1-gn are converted from digital to analog by a DAC device, such as a digital-to-analog converter packaged with flash memory, and the analog corresponding to each of the pixel values g1-gn is input to the bit lines B1-BN, and the DAC (Digital to Analog Converter) device may also be integrated directly with the flash memory on the same semiconductor die during the wafer fabrication stage. The data contained in the image gr_in can be smoothly coupled to the flash memory. Another important factor is that the final storage location or space of the image gr_in is usually necessarily various memories, so that the prominent advantage of the intrinsic characteristics that the storage location of the image gr_in and the location of performing matrix operation are the same flash memory can be revealed, the processor CPU or GPU will not need to read data from the memory to itself to perform matrix operation, the storage action and the calculation action are all implemented in the memory, and the above-mentioned problems of reading and writing data from the processor to the memory, data handling time cost and the like will not exist.

Referring to fig. 3, an important problem has not been solved, namely, the matrix ma_wei participating in the network calculation is purposely designed in order to explain the specific implementation process of the neural network. However, how to pair the m×n vector matrix ma_wei to the m×n elements of the transistors in the flash memory as shown in fig. 1 is still a challenge to be solved.

Referring to fig. 4, regarding the relationship between the pixel values g1-gn and the bit lines B1-BN, the pixel value g1 is converted into the voltage value V1 through DAC digital-to-analog conversion processing, the pixel value g2 is converted into the voltage value V2 through digital-to-analog conversion, the pixel value g3 is converted into the voltage value V3 through digital-to-analog conversion, and so on, the pixel value gn is converted into the voltage value VN through digital-to-analog conversion. According to the illustrated interface correspondence, voltage value V1 is tied to bit line B1, voltage value V2 is tied to bit line B2, voltage value V3 is tied to bit line B3, and so on. The bit line voltage value converted by the pixel with larger pixel value is larger, and the bit line voltage value converted by the pixel with smaller pixel value is lower. Furthermore, the number of transistor layouts, i.e. the size of the matrix mxn, should have a correspondence with the pixel size of the image gr_in.

Referring to fig. 4, regarding the matrix ma_wei, the respective gates of the first row transistors are connected to the word line G1 and the respective gates of the second row transistors are connected to the word line G2, and the respective gates of the mth row transistors are connected to the word line GM. The first row of transistors has their respective weights in the weight matrix denoted W ₁₁、W₁₂、W₁₃、……W_1N. The weights produced by the second row of transistors in the weight matrix by the second row of transistors, respectively, are denoted as W ₂₁、W₂₂、W₂₃、……W_2N. The weight that M-th row transistors produce in their respective M-th row in the weight matrix is denoted as W _M1、W_M2、W_M3、……W_MN. Each transistor is thus configured with a corresponding weight value that contributes to the matrix ma_wei involved in the network calculation. The conventional transistor has no feature associated with the weight itself, especially the flash memory has no direct coherence with the neural network, so how to map the weight to the transistor is another key issue to be solved.

Referring to fig. 4, the M rows and N columns of transistors in total are considered as m×n matrix elements, and the definition of the respective weight values in the matrix ma_wei is the basis for implementing the neural network NET. The source current of each row transistor is shown flowing to a source line common to the row transistors, and the drain of each column transistor is shown connected to a bit line common to the column transistors, the layout of which will give a viable solution to how to define its weight for the transistors. For example, the source current convergence characteristics of each row of transistors correspond to the multiplication and accumulation calculation algorithm of the vector matrix multiplication described above.

Referring to fig. 4, in order to explain how the weights are mapped to the transistors, a relation between the current and the threshold and the applied voltage of the memory transistor is described, that is, the threshold voltage (threshold voltage) of the gate is V _TH, the process parameter k=μcw/L of the memory transistor related to the process is the width-to-length ratio dimension of the transistor, the process parameter K is almost constant for the memory transistor which determines the manufacturing process and the determining dimension, and the voltage V _GS between the gate and the source and the voltage V _DS between the drain and the source of the transistor can be satisfied with K (V _GS-V_TH)V_DS =i. The current I flowing through the transistor is equivalent to multiplication and accumulation calculation of matrix multiplication if the current I of a single transistor is performed multiplication and the output current I of a row of transistors are performed when the transistor is operated in the linear region or triode region.

Referring to fig. 4, for the first row of transistors (gate coupled to G1), the current I1 of the first transistor is calculated to be approximately denoted as K (V _GS1-V_TH1)V_DS1 =i1, according to the same current formula, the current I2 of the second transistor is calculated to be approximately denoted as K (V _GS2-V_TH2)V_DS2 =i2, the nth transistor current K (V _GSN-V_THN)V_DSN = IN.) the voltage between the gate and source of the first transistor is V _GS1 and the voltage between the drain and source is V _DS1, the threshold voltage is V _TH1, the voltage between the gate and source of the second transistor is V _GS2 and the voltage between the drain and source is V _DS2, the threshold voltage is V _TH2, the voltage between the gate and source of the nth transistor is V _GSN and the voltage between the drain and source is V _DSN, the threshold voltage is V _THN).

Referring to fig. 4, for the first row of transistors (gate coupled to G1), the first through nth transistors each have their source current flowing to a source line S1 common to the row of transistors, the current of source line S1 being denoted IS1.

IS1=I1+I2+I3+...IN

Referring to fig. 4, for the first row of transistors (gate coupled to G1), where W ₁₁ is proportional or equal to K (V _GS1-V_TH1) and V1 is proportional or equal to V _DS1, the breakover of the current I1 through the first transistor is expressed as i1=w ₁₁ ×v1. the second transistor is located at a position where W ₁₂ is proportional to or equal to K (V _GS2-V_TH2) and V2 is proportional to or equal to V _DS2, and the breakover of the second transistor current I2 is expressed as i2=w ₁₂ ×v2. The third transistor is located at a position where if a proportional relationship or equality between W ₁₃ and K (V _GS3-V_TH3) and a proportional relationship or equality between V3 and V _DS3 are defined, the breakover of the third transistor current I3 is expressed as i3=w ₁₃ ×v3. The N-th transistor is located at a position where W _1N is proportional to or equal to K (V _GSN-V_THN) and VN is proportional to or equal to V _DSN, and the N-th transistor current IN is represented as in=w _1N ×vn.

Referring to fig. 4, for the first row of transistors (gate coupled to G1) the current of source line S1 IS expressed as IS1 IS converted by the equation can be expressed as follows (Y ₁ = IS1 can also be defined). The precondition is that the first row of transistors have their source current sink characteristics, which corresponds to a multiplication and accumulation calculation algorithm for vector matrix multiplication, and therefore belongs to one of the main factors that flash memory can replace part of the functionality of a conventional CPU or GPU.

IS1=Y₁＝W₁₁*V1+W₁₂*V2+W₁₃*V3+…W_1N*VN

Referring to fig. 4, the conversion of the current of source line S2, denoted IS2, for the second row of transistors (gate coupled to G2) can then be expressed in the following way (Y ₂ = IS2 can also be defined).

IS2=Y₂＝W₂₁*V1+W₂₂*V2+W₂₃*V3+…W_2N*VN

Referring to fig. 4, the current of the source line SM for the M-th row transistor (gate coupled to GM) is expressed as ISM-past conversion in the following manner (Y _M = ISM may also be defined).

ISM=Y_M＝W_M1*V1+W_M2*V2+W_M3*V3+…W_MN*VN

Referring to fig. 4, the definitions X ₁ to X _N are input layers of the neural network NET, and the definitions Y ₁ to Y _N correspond to output layers of the neural network NET. The first row transistor of the flash memory has completed a vector operation Y ₁ from the input layer to the output layer, and the result of Y ₁ can be directly calculated at one time by sensing the current of the source line S1 of the first row transistor, without separately calculating W ₁₁ ×v1 or W ₁₂ ×v2 or W _1N ×vn, etc., belonging to the class of calculation under the non-von neumann architecture, and the calculation result is not obtained by a numerical operation but is obtained by current sensing. Similarly, the second row of transistors of the flash memory completes one vector operation Y ₂ from the input layer to the output layer, the third row of transistors of the flash memory completes one vector operation Y ₃ from the input layer to the output layer, and the mth row of transistors of the flash memory completes one vector operation Y _M from the input layer to the output layer. In an alternative embodiment, the currents IS1-ISM are converted from analog to digital values by an ADC device, such as an analog-to-digital converter packaged with a flash memory, and the digital values corresponding to the current values are applicable to various electronic devices, and ADC (Analog to Digital Converter) devices and flash memory may also be directly integrated onto the same semiconductor die during the wafer fabrication stage.

Referring to fig. 4, one task that the transistor array of the flash memory accomplishes is to construct a matrix ma_wei, and each element in the matrix corresponds to a weight matrix of the neural network NET.

Referring to fig. 4, taking the neural network NET input layer Lay 1/hidden layer Lay2 as an example, when input data is input to the input layer Lay1, there is a matrix ma_wei between the input layer Lay1 and the hidden layer Lay2, and the input data is folded into voltages V1-VN to be given to the input layer Lay1, then the output obtained by the hidden layer Lay2 is equivalent to the operation result obtained by performing matrix operation on the input data by the weight matrix ma_wei. The voltage signal is used as input type and the output data is a current value signal, and the amplitude of the output current value is influenced by the action of the weight matrix Ma_wei on the input voltage signal, namely the approximate process of artificial intelligence operation is realized through a NOR Flash memory.

Referring to fig. 4, the flash memory is used for the above AI artificial intelligence computation, and the memory technology can be also used for other computing aspects such as brain-like chips, representing a mainstream big data computing chip architecture in the future, and can also be used as a core computing module of a low-energy-consumption super computing center. For example, the in-NOR FLASH memory computation can support the deep learning weight parameter of about 300M in a single chip, and the computation can be performed without additional memory.

Referring to fig. 4, the calculation of the neural network NET by the flash memory has several important characteristics described below compared with other types of general calculation, and also belongs to the advantages reflected by itself.

Referring to fig. 4, in a first aspect, the neural network NET calculation flow based on the flash memory is simple and allows for a large amount of parallelism. The characteristic that the calculation of the neural network NET can be effectively operated without adopting the mechanisms of branch prediction, speculative execution and the like of a general-purpose calculation conventional CPU enables the flash memory hardware accelerator for the neural network NET to be reduced or separated from the use of the CPU or the GPU as much as possible, thereby carrying out low-cost and high-efficiency calculation.

Referring to fig. 4, in a second aspect, the calculations performed by the neural network NET consist of only a few single operations, such as matrix multiplication and vector operations, the application of convolution kernels, and other linear algebraic operations. So that the neural network NET accelerator can be similar to a GPU, and can cope with complex data operation required to be executed in industry by a large number of simple calculation units.

Referring to fig. 4, in a third aspect, the neural network NET itself exhibits a very large sparsity characteristic, with a very small number of parameters and calculations, and even approaching zero. The sparsity characteristic can enable the calculation to reasonably utilize the data to skip the calculation multiplied by zero, so that the overall calculation speed is improved, and particularly matrix calculation is improved.

Referring to fig. 4, in a fourth aspect, the neural network NET allows for lower precision calculations such as low-bit, for example, for most artificial intelligence applications, 8-bit or 4-bit (meaning small transistor array) fixed-point numbers may be used for quantization with little loss of recognition accuracy, and low precision fixed-point numbers may be used for calculation according to the artificial intelligence application scenario, thereby improving the inference speed and reducing the power consumption. The power loss is a point worth emphasizing, the traditional CPU/GPU generates extremely large heat in the model training and reasoning stage, most of the power loss is in the aspect of the heat dissipation capacity of the processor, the class calculation of the application completely avoids the accumulation process of the processor aiming at the monocular multiplication and the multiplication result, reduces unnecessary data carrying links between the processor and the memory, and reduces the energy consumption to 1/10 to 1/100 of the original energy consumption.

Referring to fig. 4, for example, the intelligent voice model size is typically in the range of hundreds of K to several mega-meters, and the end-side image inference model size is typically in the range of several mega-meters to tens of mega-meters, the memory space of conventional NOR FLASH on the market can be easily adequate for such scale model volumes. Even the NOR FLASH in-memory computing chip of the handheld terminal is sufficient to meet the requirements of most AI scenes. On the premise of optimizing the NOR FLASH structure, for example, the serial-parallel connection or cascade connection of a plurality of FLASH memories is almost comparable to all artificial intelligence application scenes which can be achieved by the GPU.

Referring to fig. 5, the foregoing is explained about the proportional relationship or the equal mapping relationship between the weights, such as W ₁₁, and the transistor parameters, such as K (V _GS1-V_TH1), and in the neural network NET, the weights are information that needs to be adjusted and updated at any time, so the details of how to adaptively modify the transistor parameters (or weights) are included in the part worth explaining. In particular, neural networks include back propagation, the core of back propagation, i.e., updating and iteration of weights, it is apparent that local computation of flash memory differs from weight iteration as in conventional processors.

Referring to fig. 5, the threshold voltage (threshold voltage) of the transistor is V _TH, which if changed corresponds to K (V _GS-V_TH) being changed, equivalent to changing the weight value in proportional relation to K (V _GS-V_TH), so changing the threshold voltage V _TH is an alternative way. Writing and erasing of flash data is performed by manipulating charges inside the floating gate associated with the control gate, such as charge discharging and tunneling. For example, in the writing mode, NOR Flash adopts a hot electron injection mode, and K (V _GS-V_TH) can be changed by increasing the voltage of the control gate during writing, for example, by injecting charge into the floating gate through the source, and then changing the charge amount of the floating gate. Also, as in the erase mode, NOR Flash uses tunneling to drain the charge from the floating gate, e.g., a negative voltage is applied to the control gate.

Referring to FIG. 5, taking the M-th row (gate coupled to word line GM) transistor as an example, floating gate charge of a transistor (weight W _MN) as determined, for example, by bit line BN can be cleared during an erase operation of programming PG0, so its threshold voltage is lowered and weight W _MN is increased. The erase of program PG0 is typically performed by applying a negative voltage at word line GM to cause quantum tunneling of the floating gate charge of the transistor into the substrate on which the transistor is located. In accordance with the above-mentioned need for modification based on the weight value, instead, the floating gate charge of the transistor (weight W _MN) determined, for example, by the bit line BN can be injected in the writing action of the programming PG1, in which case its threshold voltage is increased and the weight W _MN is decreased. Further, the floating gate charge of the transistor (weight W _MN) determined by, for example, the bit line BN is continuously injected during the writing operation of the program PG2, and the program PG2 is fine-tuned with respect to the program PG1, so that the threshold voltage of the transistor is continuously increased and the weight W _MN is finely decreased. Wherein the voltage of the charge injection program PG1 is much higher than the voltage of the trimmed program PG1, the small voltage scale of the program PG2 is to avoid the weight overshoot. In essence, the charge injection programming may be performed with a greater number of injections than is shown in the figure, so that the weight W _MN may not only satisfy the majority value requirement of the weight matrix ma_wei, but also satisfy the high precision requirement of the weight value. The execution of multiple loops as shown in the erase programming action and the charge injection programming action represents an iteration of weights.

Referring to FIG. 4, in an alternative embodiment, a method of performing in-memory calculations includes utilizing an array of memory transistors in a flash memory as a carrier for a weight matrix (weight matrix) of a neural network NET, setting a series of bit lines B1-BN of the array of memory transistors (transistor arrays) to which input data of the neural network is to be supplied, and also setting output data of the neural network NET to be retrieved from a series of source lines S1-SM of the array of memory transistors. Wherein the magnitude of the weight of any transistor in the memory transistor array is adjusted by using the threshold voltage thereof, thereby dynamically configuring the weight matrix of the neural network. The weight of one transistor T ₁₁, transistor T ₁₁, as determined by bit line B1 and source line S1, is adjusted with its threshold voltage. The weight of one transistor T ₂₁, transistor T ₂₁, determined by bit line B1 and source line S2, is adjusted by its threshold voltage. The weight of one transistor T ₃₁, transistor T ₃₁, determined by bit line B1 and source line S3, is adjusted by its threshold voltage. The weight of one transistor T _M1, transistor T _M1, determined by bit line B1 and source line SM, is adjusted by its threshold voltage. The magnitude of the weight of any transistor is adjusted with its threshold voltage, thereby dynamically configuring the weight matrix (weight matrix) of the neural network. N and M are positive integers. Further, as the transistor T _MN, the weight of the transistor T _MN is adjusted by the threshold voltage of the transistor T to the floating gate charge, etc., as determined by the bit line BN and the source line SM.

Referring to FIG. 4, in an alternative embodiment, the memory transistor array (transistor arrays) for the neural network, as defined by the bit lines B1-BN and the source lines S1-SM, is located on the same physical block (block) of the flash memory, and the space in which each of the input data (e.g., vector Ma_in) and output data (e.g., vector Ma_out) of the neural network is located is separate from the memory space in which the memory transistor array (transistor arrays) is located. For example, the storage space in which the input data is located and the storage space in which the output data is located belong to the same physical block, or the storage space in which the input data is located and the storage space in which the output data is located respectively belong to the same physical block. However, the storage space where the input data and the output data are respectively located and the storage space where the storage transistor array is located are different physical blocks. The stripping of the memory space mainly takes into consideration that the memory space and the computation space are allowed to have different erasing operations and writing operations, which are equivalent to the operations of performing weight adjustment (PG 0, PG1, PG2, etc. in FIG. 5) by using threshold voltages and the operations of performing data reading and writing of memory data, such as input and output data of a neural network, in different memory spaces, the weight configuration operations and the operations of reading and writing the memory data do not interfere with each other. Because the programming voltages (or programming currents) used by the weight configuration operations and the read and write operations for storing data require different voltage levels (or current levels) to be used.

Referring to fig. 4, in an alternative embodiment, the raw input data of the neural network NET is converted into input quantities in the form of voltage values to be supplied to a series of bit lines of the memory transistor array based on the input quantities ordered into vectors. The raw input data, for example, includes data types such as pixel values or gray values of an image or video, and is converted into input quantities V1, V2, V3.. The voltage V1 is input to the bit line B1, the voltage V2 is similarly input to the bit line B2, the voltage V3 is input to the bit line B3, and the voltage VN is input to the bit line BN.

Referring to fig. 4, the input is represented by vector ma_in= [ b1=v1, b2=v2..bn=vn ]. The series of bit lines B1-BN are delivered to the array (transistor arrays) in an input quantity [ V1, V2..VN ] based on the ordered vector.

Referring to fig. 4, in an alternative embodiment, the output data of the neural network NET is characterized by the output currents of the memory transistor array, the output currents of a series of source lines of the memory transistor array being regarded as the vectorized output quantity of the memory transistor array. The series of source lines S1-SM, e.g., memory transistor array (transistor arrays), have their respective output currents respectively denoted IS1, IS 2. The output current IS1 of the source line S1 may be analog-to-digital converted by an ADC and the output current ISM of the similar source line SM may be analog-to-digital converted by an ADC. The output data of the network NET IS characterized by the output currents IS1, IS2, ISM of the memory transistor array.

Referring to fig. 4, in an alternative embodiment, the output currents of the series of source lines S1-SM of the array of storage transistors are considered to be the output quantities of the array of storage transistors ordered into a vector ma_out [ IS1, IS2,..ism ]. The output of the neural network can also be considered as represented by vector ma_out= [ s1=is1, s2=is2,..sm=ism ].

Referring to fig. 5, in an alternative embodiment, the manner in which the weights of the transistors (T _MN as determined by BN/SM) are adjusted includes changing the floating gate injection charge (e.g., changing the floating gate charge of T _MN), by which the magnitude of the weight of any transistor at the location in the storage transistor array (e.g., the location as determined by BN/SM) is adjusted, e.g., the magnitude of the weight W _MN of T _MN.

Referring to fig. 5, in an alternative embodiment, the weight matrix of the neural network is dynamically configured by resetting the weight sizes at various locations in the memory transistor array during the back propagation phase of the neural network NET. The various locations in the array referred to herein include each where the bit lines B1-BN and source lines S1-SM intersect each other, and in fact represent the column and row indices of the various elements in the weight matrix. The column index of this element, e.g., W _MN, in the weight matrix is the position determined or corresponding by the bit line BN/source line SM. The neural network needs to iterate the magnitude values of each element in the weight matrix repeatedly in the back propagation algorithm until the magnitude values meet the requirements.

Referring to FIG. 4, in an alternative embodiment, input data of the neural network is supplied to a series of bit lines of the array of memory transistors and output data of the neural network is retrieved from a series of source lines of the array of memory transistors, wherein the source of each row of memory transistors in the array of transistors is connected and considered as a source line output of one output data of the neural network, and the drain of each column of memory transistors in the array of transistors is connected and considered as a bit line input of one input data of the neural network. In addition, the gates of each row of memory transistors in the memory transistor array are connected to couple to a word line.

Referring to fig. 3, in an alternative embodiment, a digital-to-analog converter DAC is provided at each of the bit lines B1-BN of the memory transistor array for converting the input data of the neural network NET into voltage values matching the input signal pattern of the bit lines (e.g., the bit line required voltage signal as an input quantity).

Referring to fig. 3, in an alternative embodiment, an analog-to-digital converter ADC is configured at each of the source lines S1-SM of the array of storage transistors for converting the output signal in the output current format at the source line into output data adapted to the digital output quantity of the neural network NET. For example an analog-to-digital converter arranged at the source line SM for converting the output signal in the form of an output current ISM at the source line SM into output data adapted to the digital output of the neural network. The digital quantity (digital) of the current is different from the analog quantity (analog). The analog-to-digital converter configured at the source line S1 IS used to convert the output signal in the form of the output current IS1 at the source line S1 into output data adapted to the digital output of the neural network. The same is true for the other S2 to SM-1.

Referring to fig. 4, in an alternative embodiment, the means for performing in-memory calculations includes a flash memory and an input unit, an output unit, and the like. With respect to the flash memory portion of the device, the memory transistor array thereof may be used as a carrier for a weight matrix of the neural network, setting a series of bit lines for the input data of the neural network to be supplied to the memory transistor array and setting output data to be retrieved from a series of source lines of the memory transistor array to the neural network. And with respect to the input cell portion in the device, raw input data of the neural network can be converted into input quantities in the form of voltage values by the input cell portion to be supplied to a series of bit lines of the memory transistor array based on the input quantities ordered into vectors. With respect to the output unit of the device, for converting the output signal in the output current format of the source lines into output data adapted to the digital output quantity of the neural network, the output currents of a series of source lines of the memory transistor array are regarded as the output quantity of the memory transistor array ordered into vectors. The input unit and the output unit respectively comprise a digital-to-analog converter and an analog-to-digital converter.

Referring to fig. 4, in an alternative embodiment, a memory system performing in-memory calculations includes a memory chip based on flash memory and a digital-to-analog converter built into the memory chip and an analog-to-digital converter built into the memory chip. A memory chip based on flash memory, wherein the memory transistor array is used as a carrier of a weight matrix of a neural network in the flash memory, and the size of the weight of any transistor is adjusted by using the threshold voltage of the memory transistor array, thereby dynamically configuring the weight matrix of the neural network, setting a series of bit lines for transmitting input data of the neural network to the memory transistor array, and also setting output data retrieved from a series of source lines of the memory transistor array. Regarding the digital-to-analog converter built in the memory chip, the digital-to-analog converter is configured at each bit line of the memory transistor array, so that the input data of the neural network can be converted into a voltage value matched with the input signal pattern of the bit line. With respect to an analog-to-digital converter built in a memory chip, an analog-to-digital converter is configured at each source line of a memory transistor array to convert an output signal in an output current format at the source line into output data adapted to a digital output quantity of a neural network.

Referring to fig. 5, in the conventional electronic device, the calculation and the storage are separated, and the calculation, for example, the processor and the storage, for example, the hard disk or the memory, are respectively focused on the present task, and the calculation and the storage are basically not interfered with each other. There are also manufacturers that integrate processors and memories in close physical locations for increasing computing speed and computing power, for example, in circuit board layout designs where they are close together, short circuits, and are stacked directly on top of each other during the packaging stage.

Referring to fig. 5, the computational force boost has been advancing stepwise in accordance with moore's law, but the dominant computational paradigm has followed the von neumann architecture. As a new computing power, the industry mainly focuses on the narrow computing integration, that is, computing by using a storage medium, but the computing medium method is not completely unified. The integrated calculation is used as novel calculation power, and is hopeful to solve the problems of compiling walls, storage walls and power consumption walls under the traditional von neumann architecture. There is a consensus in the industry that it has basically been determined as one of the key technologies to solve the computational problem.

Referring to fig. 5, the input data or output data of the neural network mapped to the memory transistor array of the present application may be data of a certain layer (input layer/hidden layer/output layer, etc.) of the neural network, in other words, the memory transistor array is allowed to be only a local sub-network structure of the neural network. Alternatively, the neural network may use a plurality of memory transistor arrays as shown to perform serial-parallel cascade combination of the front and rear stages, and the plurality of memory transistor arrays may be combined into a local sub-network structure of the neural network or into an overall network structure.

Referring to fig. 5, the storage and calculation integration organically integrates the storage and the calculation, and is expected to become an advanced calculation architecture scheme in the computing age based on the huge energy efficiency ratio and the promotion potential. There is a doubt about how to cope with the weight shift (for example, shift occurring in W _MN) encountered in updating the weights of the internal neural network of the memory, which is a difficult problem of imperceptibility for the memory, and will be explained in detail further below.

Referring to FIG. 6, it is known that a memory transistor array (transistor arrays) for a neural network NET can be defined by bit lines B1-BN and source lines S1-SM, which array is taken as an example to illustrate the problem of weight shift. The size of the memory transistor array should be matched with the actual application scene of the neural network, for example, the neural network often needs to perform convolution calculation on images with various length, width and size to obtain a convolution result. For example, even a simple 28 x 28 long by 28 wide handwriting recognition image may require the provision of approximately 784 input ports, in which case the size of the memory transistor array is quite large, as is apparent. A troublesome problem is that the length of the word line becomes very long with the increasing size of the memory transistor array, because the adjustment of the weight is dependent on the programming of the charge writing or erasing of the floating gate of the transistor by the word line, and the excessively long word line causes a very large word line parasitic capacitance C _P on the transistor array, resulting in an error in the weight programming. The parasitic capacitances include at least the gate-source and gate-drain parasitic capacitances of the individual transistors coupled to the word line and the capacitance of the word line to the substrate. Memory transistor arrays are fabricated on semiconductor substrates. Programming errors also mean deviations in the weight configuration. It should be noted that in many neural networks, such as convolutional neural networks in the intelligent driving domain video sensing, the input image resolution related to the number of input ports of the transistor array is 1628×1236, 1920×1080, 2048×1024, 3840×2160, and the like, whether the image length and width are scaled or not is accompanied by a huge array size, and the parasitic capacitance C _P is complicated, because the size of the parasitic capacitance and the size of the array are not completely corresponding. Parasitic capacitance has both parallel and series capacitances, such as the body capacitance at the transistors, and also coupling capacitances, such as parasitic coupling capacitances between different word lines, which gives a significant error rate for weight iterations, such as back propagation algorithms, and there is a significant irrational in training and reasoning. This is not a concern in the conventional application scenario of memory, as memory confronts oversized or mass data storage applications that can handle this storage requirement in partitioned physical blocks Blokc or planar planes or granular LUNs, but in the neural network scenario weight element programming condition inconsistencies tend to be severe if the weight matrix is partitioned into different physical regions.

Referring to fig. 6, in an alternative embodiment, for any transistor on any word line of the memory transistor array during the phase of dynamically configuring weights (the dynamically configuring weights are because the parameters of the neural network are continuously iterating), the any transistor first loads a reverse voltage transient of the programming voltage used by other transistors on the any word line adjacent to the previous weight adjustment onto the any word line before performing the any weight adjustment, and then applies the programming voltage set for the any transistor onto the any word line to perform the any weight adjustment. One of the purposes is to suppress the weight update iteration failure caused by parasitic charge of the parasitic capacitance present at the any one word line of the memory transistor array. Parasitic charge is typically affected by the survivors of the programming voltages of the other transistors on either word line that perform weight adjustment. Assuming that the training number epoch=5000 of the expected iterations of the neural network in high image resolution conditions, the parasitic charge of any one word line will be quickly hidden out of control with the superposition of such a large array size and so many iterations, making it difficult to grasp the true value of the actual parasitic charge of the word line and, if known, be quickly refreshed by the next iteration.

Referring to fig. 6, in an alternative embodiment, the issue of how to better cope with the weight offset faced by the weight update of the internal neural network in the memory is that the basic scheme is to adjust the weight of any transistor in the memory transistor array by using its threshold voltage, so as to dynamically configure the weight matrix of the neural network. A further optimization is directed to a stage of dynamically configuring weights for any transistor on any word line of the memory transistor array, wherein the any transistor first loads a reverse voltage transient of a programming voltage used by other transistors adjacent to a previous weight adjustment on the any word line onto the any word line before executing the any weight adjustment, and then applies the programming voltage set for the any transistor onto the any word line to execute the any weight adjustment.

Referring to fig. 6, for example, the weight of transistor T _MN in the transistor array (transistor arrays) is adjusted by its threshold voltage V _THMN, transistor T _MN is one transistor determined by bit line BN and source line SM and the word line it is in is GM. The positive integer m=1, 2, 3 is set. Positive integer n=1, 2, 3. Whereby the weight matrix ma_wei of the dynamically configurable neural network NET, V _THMN, V _TH is the threshold and the subscript MN is the index and it indicates that the threshold voltage of any transistor and the weight W _MN can be manipulated by the index position.

Referring to fig. 6, for example, in the stage of dynamically configuring the corresponding weight W _MN for any transistor, e.g., T _MN, on any word line, e.g., on the word line GM, the transistor of the adjacent previous weight adjustment on the word line, e.g., the transistor of the adjacent previous weight adjustment on the word line GM, e.g., the transistor of T _M3 on the GM is assumed to be the transistor of the adjacent previous weight adjustment on the GM on which T _MN performs the current weight adjustment, e.g., epoch=1455, is preferably mounted on any word line GM with a reverse transient voltage, e.g., V _PGM3, before any weight adjustment is performed, e.g., the number of times epoch=1455. After that, a programming voltage such as V _PGMN set for the transistor T _MN on the arbitrary transistor such as GM is applied to, for example, the word line GM to perform the adjustment operation of the weight W _MN of the arbitrary weight adjustment such as the number of epoch=1455. The weight (e.g., W _MN) update iteration that suppresses the parasitic charge of the parasitic capacitance present at either word line GM of the memory transistor array fails and it is worth emphasizing that this scheme is not suitable for in-memory data storage applications, which would otherwise have a high probability of causing an unexpected inversion of the logic value of the stored data bit. Note that the parasitic charge of the word line GM has a volatile characteristic but the power loss rate of such a volatile may be low relative to the speed of the weight matrix iteration, which is related to the transistor layout density of the memory transistor array, the choice of materials for the wiring and other components, and the critical dimensions of the transistors, etc. The error rate of the weights can be compensated and saved as much as possible by the scheme, and the error rate is complemented with the perfection and training of the parameters of the neural network such as the weights. An important goal of the foregoing scheme is also to compress the step difference between the programming voltage set by any one of the transistors performing any one weight adjustment and the programming voltages used by other transistors on any one of the word lines that are adjacent to the previous weight adjustment, avoiding any coupling of the programming voltages set by any one of the transistors performing any one weight adjustment to the adjacent word line of any one of the word lines. Thereby preventing undesired iterative changes in the weight values of portions of the transistors on adjacent word lines.

Referring to fig. 6, for example, the foregoing scheme greatly compresses the step difference (voltage step) between the programming voltage set by any one time of weight adjustment by any one transistor such as T _MN, such as V _PGMN, and the programming voltage V _PGM3 used by other transistors adjacent to the previous time of weight adjustment on any one word line such as GM, such as transistor T _M3, avoiding any transistor such as transistor T _MN from performing weight adjustment any one time (e.g., for an epoch=1455) such as voltage V _PGMN being coupled to the adjacent word line GM-1 or gm+1 of any one word line GM, etc. If the voltage V _PGMN is coupled to GM-1 or GM+1 or the like in the memory transistor array (transistor arrays), the weight values of some of the transistors coupled on the word line GM-1 or GM+1 (e.g., where GM-1/GM+1 intersects BN) may be inadvertently actively altered non-iteratively and such that the weight matrix is locally rather than globally modified in a hidden manner. In this case the neural network produces erroneous results, whether it is training or reasoning. The scheme of the application skillfully solves the problem that the weight values of part of transistors on adjacent word lines are changed in unexpected iteration.

Referring to fig. 6, based on the foregoing, the convolutional neural network is trained with training samples, and the trained parameters such as weight matrix and the like have good generalization performance for the test samples. The method has the advantages of considerable robustness and recovery of unexpected offset in terms of resisting error weight caused by parasitic capacitance on any word line in the memory transistor array. Belongs to a convolutional neural network or a subnetwork with good performance. Neural networks are typically adapted for intelligent speech models, end-side image inference models, and most other AI artificial intelligence scenarios.

The foregoing description and drawings set forth exemplary embodiments of the specific structure of the embodiments, and the foregoing invention provides presently preferred embodiments, without being limited to the precise arrangements and instrumentalities shown. Various alterations and modifications will no doubt become apparent to those skilled in the art after having read the above description. Therefore, the appended claims should be construed to cover all such variations and modifications as fall within the true spirit and scope of the invention. Any and all equivalent ranges and contents within the scope of the claims should be considered to be within the intent and scope of the present invention.

Claims

1. A method of performing in-memory computation, comprising:

Setting a series of bit lines of the neural network, which are used for transmitting input data of the neural network to the memory transistor array, and setting output data of the neural network, which are extracted from a series of source lines of the memory transistor array, by using the memory transistor array as a carrier of a weight matrix of the neural network in the flash memory, wherein the weight of any transistor in the memory transistor array is adjusted by using the threshold voltage of the transistor array, so that the weight matrix of the neural network is dynamically configured;

In the stage of dynamically configuring the weight for any transistor on any word line of the memory transistor array, before any weight adjustment is executed, the any transistor firstly carries out transient mounting of a reverse voltage of a programming voltage used by other transistors adjacent to the previous weight adjustment on any word line once, and then applies the programming voltage set for any transistor on any word line to execute the any weight adjustment, so that weight update iteration failure caused by parasitic charge of parasitic capacitance existing at any word line of the memory transistor array is restrained.

2. The method according to claim 1, characterized in that:

The memory transistor array for the neural network being located on the same physical block of the flash memory, and

The storage space where the input data and the output data of the neural network are respectively located and the storage space where the storage transistor array is located respectively belong to different physical blocks of the flash memory.

3. The method according to claim 1, characterized in that:

the raw input data of the neural network is converted into input quantities in the form of voltage values for delivery to a series of bit lines of the memory transistor array based on the input quantities ordered into vectors.

4. The method according to claim 1, characterized in that:

The output data of the neural network is characterized by the output currents of the memory transistor array, the output currents of a series of source lines of the memory transistor array being regarded as the vector ordered output of the memory transistor array.

5. The method according to claim 1, characterized in that:

the manner in which the weight of any transistor is adjusted includes changing its floating gate injection charge, by which the magnitude of the weight of any transistor at the location in the array of storage transistors is adjusted.

6. The method according to claim 5, wherein:

In the back propagation phase of the neural network, the weight matrix of the neural network is dynamically configured by resetting the weight magnitudes at various locations in the memory transistor array.

7. A method according to claim 3, characterized in that:

A digital-to-analog converter is provided at each bit line of the memory transistor array for converting input data of the neural network into voltage values matching the input signal pattern of the bit line.

8. The method according to claim 4, wherein:

An analog-to-digital converter is provided at each source line of the array of storage transistors for converting the output signal in output current format at the source line into output data adapted to the digital output of the neural network.

9. An apparatus for performing in-memory calculations, comprising:

The flash memory takes the storage transistor array as a carrier of a weight matrix of the neural network, sets input data of the neural network to be transmitted to a series of bit lines of the storage transistor array, and sets output data of the neural network to be extracted from a series of source lines of the storage transistor array, wherein the weight of any transistor in the storage transistor array is adjusted by utilizing a threshold voltage of the transistor, so that the weight matrix of the neural network is dynamically configured, and the weight updating iteration failure caused by parasitic charge of parasitic capacitance existing at any word line of the storage transistor array is restrained for the stage of dynamically configuring the weight of any transistor on any word line of the storage transistor array, wherein the reverse voltage transient of the programming voltage used by other transistors adjacent to the previous weight adjustment on any word line is firstly hung on the any word line once before any weight adjustment is executed, and then the programming voltage set for any transistor is hung on any word line to execute any weight adjustment;

10. A storage system for performing in-memory calculations, comprising:

Setting a series of bit lines of the input data of the neural network to the memory transistor array and also setting output data captured from a series of source lines of the memory transistor array to the neural network, wherein in the stage of dynamically configuring weights for any transistor on any word line of the memory transistor array, before any weight adjustment is performed, the any transistor transient mounts a reverse voltage of a programming voltage used by other transistors adjacent to the previous weight adjustment on any word line once to the any word line, and then applies the programming voltage set for the any transistor on the any word line to perform any weight adjustment, thereby inhibiting weight update iteration failure caused by parasitic charge of parasitic capacitance existing at any word line of the memory transistor array;