KR20190008514A - Natural language processing with knn - Google Patents

Abstract

A system for processing natural language comprises a memory array and a processor. The memory array is divided into a similarity section for storing a plurality of feature vectors, a SoftMax section for determining the probability of generation of the feature vectors, a value section for storing a plurality of modified feature vectors, and a marker section. In each column displayed by the marker section, the processor activates the memory array to perform, in parallel: similarity calculation in the similarity section between the feature vector, stored in the displayed column, and a vector question; SoftMax calculation in the SoftMax section for determining a SoftMax probability value related to the displayed feature vector; multiplication calculation in the value section for multiplying the related SoftMax probability value by each modified feature vector stored in the displayed column; and vector sum calculation in the value section for accumulating attention vectors of output of the multiplication calculation. The present invention is to analyze a large data set in a very rapid and efficient manner.

Description

Natural language processing using KNN {NATURAL LANGUAGE PROCESSING WITH KNN}

Cross-reference to related application

This application claims priority to U.S. Provisional Application No. 62 / 533,076, filed July 16, 2017, and U.S. Provisional Application No. 62 / 686,114, filed on June 18, 2018, the contents of which are incorporated herein by reference in their entirety do.

Technical field

The present invention relates generally to data mining algorithms that use association calculations, and in particular association calculations.

Data mining is a computation process that finds patterns in large data sets. It analyzes the data set using different techniques. One such technique is classification, a technique used to predict the group membership of a new item based on data associated with an item in a data set for which the group member is known. Although the k-nearest neighbors (k-NN) algorithm is not limited to this, among many other applications, machine learning procedures such as bioinformatics, speech recognition, image processing, statistical estimation, It is one of the known data mining classification methods used in many fields.

In a large data set of objects (e.g., product, image, face, voice, text, video, human condition, DNA sequence, etc.), each object may be associated with one of a number of predefined classes For example, a product class can be a clock, a vase, an earring, a pen, etc.). The number of classes may be small or large, and besides being associated with a class, each object may be described by a set of attributes (e.g., size, weight, price, etc. for products). Each of the attributes may be further defined by a numerical value (e.g., in the case of product size: e.g., 20.5 cm wide). The purpose of the classification procedure is to identify classes of unclassified objects (classes are not yet defined) based on the values of the attributes of the objects and the similarity of the data sets to the pre-classified objects.

The K-nearest neighbors algorithm first calculates the similarity between the introduced object X (unclassified) and every individual object in the data set. Similarity is defined as the distance between objects so that the smaller the distance, the more similar the object is, and there are some known distance functions that can be used. After the distance is computed between the new introduced object X and all objects of the data set, k nearest neighbors for X can be selected, where k is a predefined definition defined by the user of the K-nearest neighbor algorithm . X is assigned to the most common of the k nearest neighbors.

Among other algorithms, the K-nearest neighbors algorithm must analyze very large and unsorted large data sets very quickly and efficiently to quickly access the smallest or largest, or extreme, k items of the data set.

One way to find these k smallest / largest items in a data set may be to sort the data sets first so that the numbers are arranged in order and the first (or last) k number is the desired k item in the data set . Numerous alignment algorithms are known and can be used in the art.

One in-memory alignment algorithm is described in US patent application Ser. No. 14 / 594,434, filed January 1, 2015, and assigned to a co-assignee of the present application. The algorithm first searches for a first minimum (or maximum), then a second minimum (or maximum), and then until all the numbers in the data set are sorted from minimum to maximum (or maximum to minimum) It can be used to sort the number of sets by repeating the process. The computational complexity of the sorting algorithm described in US patent application Ser. No. 14 / 594,434 is O (n) when n is the size of the set (since there are n iterations to sort the entire set). If the calculation is stopped at the kth iteration (if used to find the first k min / max), the complexity may be O (k).

According to a preferred embodiment of the present invention, a system for natural language processing is provided. Such systems include memory arrays and in-memory processors. The memory array has rows and columns and includes a similarity section for initially storing a plurality of keys or feature vectors, a SoftMax section for determining the probability of occurrence of a key or feature vector, a plurality of modified feature vectors for initial storage A value section, and a marker section. An operation on one or more columns of the memory array is associated with a feature vector to be processed. The in-memory processor is in each column indicated by the marker section:

Calculating a similarity in a similarity section between each feature vector and a vector query stored in each displayed column;

A SoftMax operation in the SoftMax section that determines the associated SoftMax probability value for each displayed feature vector;

A multiplication operation in a value section that multiplies the associated SoftMax probability value by each modified feature vector stored in each displayed column; And

The vector sum in the value section, which accumulates the attention vector sum of the output of the multiplication operation - the vector sum will be used to generate a new vector query for further iteration or to generate an output value in the final iteration -

In parallel.

Further in accordance with a preferred embodiment of the present invention, the memory array comprises a plurality of arithmetic units, one for each iteration of a natural language processing operation, and each arithmetic unit is divided into sections.

Moreover, in accordance with a preferred embodiment of the present invention, the memory array is an SRAM, non-volatile, volatile, or non-destructive array.

Further in accordance with a preferred embodiment of the present invention, the memory array includes a plurality of bit line processors, one for each column of each section, and each bit line processor operates on one bit of data of the associated section.

Additionally in accordance with a preferred embodiment of the present invention, the system further comprises a neural network feature extractor for generating feature vectors and modified feature vectors.

Furthermore, according to a preferred embodiment of the present invention, the feature vector comprises a word, a sentence, or a feature of a document.

According to a preferred embodiment of the present invention, the feature vector is an output of a pre-trained neural network.

Furthermore, in accordance with a preferred embodiment of the present invention, the system further comprises a pre-trained neural network for generating an initial vector query.

Further in accordance with a preferred embodiment of the present invention, the system further comprises a query generator for generating an additional query from the initial vector query and the attack vector sum.

Furthermore, according to a preferred embodiment of the present invention, the query generator is a neural network.

Alternatively, in accordance with a preferred embodiment of the present invention, the query generator is implemented as a matrix multiplier on the bit line of the memory array.

According to a preferred embodiment of the present invention, there is also provided a method for natural language processing. The method includes the steps of: providing a memory array having rows and columns, the memory array comprising: a similarity section for initially storing a plurality of keys or feature vectors; a SoftMax section for determining a probability of occurrence of a key or feature vector; A value section for initially storing the modified feature vector, and a marker section, wherein the operation in one or more columns of the memory array is associated with a feature vector to be processed; And activating the memory array to perform the following operations in parallel in each column represented by the marker section. These operations include: calculating a similarity in a similarity section between each feature vector and a vector query stored in each displayed column; A SoftMax operation in the SoftMax section that determines the associated SoftMax probability value for each displayed feature vector; A multiplication operation in a value section that multiplies the associated SoftMax probability value by each modified feature vector stored in each displayed column; And a vector sum operation in the value section that accumulates the sum of the affine vectors of the outputs of the multiplication operations. The vector sum is used to generate a new vector query for additional iteration or to generate an output value at the final iteration.

Further in accordance with a preferred embodiment of the present invention, the memory array comprises a plurality of bit line processors, one for each column of each section, and the method further comprises the step of each bit line processor operating on one bit of data of the associated section .

Furthermore, according to a preferred embodiment of the present invention, the method further comprises generating a feature vector and a modified feature vector using a neural network and storing the feature vector and the modified feature vector in a similarity section and a value section, respectively.

And in accordance with a preferred embodiment of the present invention, the method further comprises generating an initial vector query using a pre-trained neural network.

Additionally in accordance with a preferred embodiment of the present invention the method further comprises generating an additional query from the initial vector query and the attack vector sum.

Further in accordance with a preferred embodiment of the present invention, generating additional queries utilizes a neural network.

Finally, in accordance with a preferred embodiment of the present invention, generating an additional query includes performing a matrix multiplication on the bit line of the memory array.

The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings, in which:
Figures 1a and 1b are logical schematic and physical schematic diagrams of a memory computing device for constructing and operating in accordance with a preferred embodiment of the present invention to calculate k extremum values within a given time.
2 is a schematic diagram of a data set C stored in a memory array;
3 is an example of a data set C;
Figures 4 and 5 are schematic diagrams of the temporary storage used in the calculation;
6 is a flow chart illustrating the calculation steps of a k-min (k-Mins) processor;
Figures 7-11 are illustrative diagrams of the calculation steps of a k-minimum processor, constructed and operative in accordance with a preferred embodiment of the present invention, for the exemplary data set of Figure 3;
12 is a schematic diagram of one embodiment of an efficient shift for use in a count operation used by a k-minimum processor;
Figure 13 is a schematic of the event flow of numerous data mining instances;
14 is a schematic illustration of a memory array having a plurality of bit line processors;
15 is a schematic diagram of an associated memory layout operative and configured to implement an end-to-end memory network for natural language processing;
16 is a schematic diagram of an associated processing unit for implementing all the hops of the network in memory within a certain time.
It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Also, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

Applicants have realized that the complexity of the known alignment mechanism is proportional to the size of the data set, so it is not efficient to sort the data set to find the k-minimum value when the data set is very large. As the data set grows, the effective time to respond to a request to retrieve the k-min value from the data set will increase.

Applicants can use an associative memory device to store a large data set, which is not the size of the data set itself, but any size of data (e.g., data) having a constant computational complexity (O (1)) proportional to the size of the object of the data set In-memory method of finding a k-minimum value in a set.

A memory device capable of providing this constant complexity is disclosed in U.S. Patent Application 12 / 503,916, filed July 16, 2009, now U.S. Patent No. 8,238,173; U.S. Patent Application 14 / 588,419, filed January 1, 2015; U.S. Patent Application 14 / 594,434, filed January 12, 2015 (registered as U.S. Patent No. 9,859,005); U.S. Patent Application No. 14 / 555,638, filed November 27, 2014 (registered as U.S. Patent No. 9,418,719); And U.S. Patent Application No. 15 / 146,908, filed May 5, 2016, now U.S. Patent No. 9,558,812, all of which are assigned to the common assignee of the present invention.

Applicants have also realized that associative computation can provide a fast and efficient method of finding the k-minimum value with minimum latency per request, with constant computational complexity. Also, the data in the associative memory is not moved during computation, but may remain in the original memory location before computation.

It will be appreciated that increasing the data set size may not affect the computational complexity and response time of the k-minimum query.

Reference is now made to Figs. 1A and 1B, which are schematic diagrams of a memory computing device 100 constructed and operative in accordance with a preferred embodiment of the present invention. 1A, a device 100 includes a memory array 110 for storing a data set, a k-minimum processor 120 implemented on a memory logic element to perform k-minimum operation, Minimum temporary store 130 that may be used to store intermediate and final results of operations performed by k-minimum processor 120 on data stored in memory 110. [ In FIG. 1B, the physical aspects of k-minimum processor 120 and k-minimum temporary storage 130 are shown in associative memory array 140. The associative memory array 140 combines the operation of the k-minimum processor 120 with the storage of the k-minimum temporary storage 130. The memory array 110 may store a very large data set of binary numbers. Each binary number consists of a fixed number of bits and is stored in a different column of the memory array 110. The k-minimum temporary store 130 stores multiple copies of the information stored in the memory array 110 and temporary vectors associated with computation steps performed by the k-minimum processor 120, Lt; RTI ID = 0.0 > k < / RTI >

It will be appreciated that the data stored in the memory array 110 and the associated memory array 140 (to enable the execution of Boolean operations as described in the aforementioned U.S. patent application) may be stored in columns. However, for clarity, the description and drawings provide a logical view of the information whose numbers are displayed horizontally (in rows). It will be appreciated that the actual storage and computation is done vertically.

로 구성되며, 여기서

는 행 p에 저장된 이진수의 비트 i를 나타낸다. m(이진수를 포함하는 비트 수)의 값은 8, 16, 32, 64, 128 등일 수 있다.2, which is now referenced, is a schematic diagram of a data set C stored in memory array 110. [ As described above, the rows of data set C are stored as columns in the memory array 110. Data set C can store multi-bit binary numbers in q rows. Each of the binary number of the data set C is referred to as C ^P, where p is the row identifier of the memory ereoyi C is a binary number is stored. Each number C ^P is m bits

≪ / RTI >

Represents the bit i of the binary number stored in row p. The value of m (the number of bits including binary numbers) may be 8, 16, 32, 64, 128, and so on.

는 어레이 C의 셀(행 p와 열 i의 교집합)을 나타내며, 여기서 (p = 1...q; i = 1...m)이다. 도 2에서 행 3 열 2에 있는,

로 언급되는 아이템이 사각형으로 표시되어 있다.As described above, C ^P represents the row p of the array C, where (p = 1 ... q) and C _i represents the column _i of the array C, where i = 1 ... m)

(P = 1 ... q; i = 1 ... m) of the array C (intersection of row p and column i). In Figure 2, row 3 column 2,

Quot; is indicated by a rectangle.

3, which is now referenced, is an example of a data set C with eleven binary numbers, q = 11. Each row is labeled with an identifier ranging from 0 to 10. The binary numbers of the exemplary data set C each have 8 bits, and the bits are stored in a column labeled as bits 7 through 0, where m = 8 in this example. The decimal value of each binary number is shown to the right of each row. The desired amount of minimum binary numbers to be found in this example may be set to 4, i.e. k = 4, and the four smallest numbers in the data set of Figure 3 are (a) the number 14 stored in row 9; (b) number 56 stored in row 5; (c) number 88 stored in row 1; And (d) the number 92 stored in row 4.

은 0이다 - 을 선택하여 다음 단계로 계속 진행함으로써 k-최소치 세트를 생성할 수 있다. 특정 위치( i번째 비트)에서 값 0을 갖는 이진수는 동일한 위치에서 값 1을 갖는 이진수보다 더 작음을 알 수 있다.The k-minimum processor 120 constructed and operative in accordance with the preferred embodiment of the present invention can find k smallest binary numbers in a large data set C. The k smallest number of groups in data set C are referred to as k-min set and may have k numbers. k-minimum processor 120 scans column C _i of data set C from MSB (most significant bit) to LSB (least significant bit), and concurrently reads row C ^P -

Is set to 0, and then proceed to the next step to generate a k-min set. It can be seen that a binary number having a value of 0 at a particular position (i-th bit) is smaller than a binary number having a value of 1 at the same position.

The amount of the selected row is compared with the target row k. If the amount of the selected row is greater than k, the k-minimum processor 120 may continue to scan the next bit of the already selected row because there are too many rows, and the set will be further reduced. (Unselected rows may contain binary numbers with larger values, so they are not considered in the remainder of the calculations). If the amount of the selected row is less than k, the k-minimum processor 120 may add the selected row to the k-minimum set and continue to scan the next bit of all remaining binary numbers. (Since the amount of the selected row is not sufficient, additional, more binary rows will be considered). If the amount of the selected row is exactly k, the k-minimum processor 120 may stop processing because it may contain k items that the k-minimum set requires.

When k = 1, it can be seen that the k-minimum set includes a single number that is the global minimum of the entire data set. It will also be appreciated that the data set may have more than one instance with this value, and that the first instance of this value is selected as a member of the k-minimum set.

It will also be appreciated that the k-minimum processor 120 can be constructed with information stored in the memory array 110 with the bits of the binary number of the data set C. In the example of FIG. 3, the binary number is represented by a row, where MSB is the leftmost bit, LSB is the rightmost bit, and all other bits are in between. The arrangement of the binary numbers of the memory array 110 is such that the bits at the i-th position of all binary numbers of the data set C are located in the same row C _i in the memory array 110. That is, the MSBs of all binary numbers in data set C may be in the same row, and the LSBs of all binary numbers in data set C may be in the same row, thus all bits in between.

4 and 5, which are now referenced, are schematic diagrams of k-minimum temporary storage 120 constructed and operative in accordance with a preferred embodiment of the present invention. The k-minimum temporary storage unit 120 may include intermediate information stored in a vector. The vector used by the k-minimum processor 120 is vector D - a temporary inverse vector; Vector V-qualification k-minimum marker vector; Vector M - candidate vector; Vector N - Temporary Candidate Vector; And a vector T - a temporary member vector. The size (number of rows) of all vectors used in the k-minimum section 120 is q and is equal to the number of rows in data set C. [ Each vector stores, in each row, an indication associated with the binary number stored in the associated row of data set C in relation to the k-minimum set, such as a portion of the set, a candidate to join the set, and so on. It will be appreciated that the vector is physically stored in a row of the memory array 110, such as the entire data set, but is shown as a column for clarity.

Vector D is a temporary inverse vector that may contain the inverse value of the bits of column C _i processed by the k-minimum processor 120. As described above, the bits of the binary number of the data set C may be processed from the MSB to the LSB, and at each step, the k-minimum processor 120 may process another row i of the memory array 110 .

Vector D is the inverse of the column C _i processed in data set C:

D = NOT C _i

(데이터세트 C의 행 p)에 저장된 원래 비트의 값이 0이었음을 나타낼 수 있으며, 이는 데이터세트 C의 행 p에 저장된 이진수가 k-최소치 세트에 참여할 후보가 될 수 있음을 나타낸다. 유사하게, 0의 값을 갖는 벡터 D내의 모든 행 p(즉, D^P = 0)는 셀

(데이터세트 C의 행 p)에 저장된 원래 비트의 값이 1이었음을 것을 나타낼 수 있고, 이는 데이터세트 C로부터의 관련 이진수가 평가되는 데이터세트로부터의 다른 수보다 크기 때문에 k-최소치 세트에 참여할 후보가 아닐 수 있음을 나타낸다.Any row p (i.e., D ^P = 1) of the vector D having a value of one

(The row p of the data set C) was zero indicating that the binary number stored in row p of data set C can be a candidate to participate in the k-minimum set. Similarly, all rows p (i. E., D ^P = 0) in a vector D having a value of zero

(The row p of the data set C) is 1, which is larger than the other number from the data set from which the relevant binary number from the data set C is to be evaluated, so that the candidate to participate in the k- .

The vector V is a qualification k-minimum marker vector and maintains a list of all rows of data set C with binary numbers that are part of the (already) k-min set. As with the other vectors used by the algorithm, it is a q-sized vector that holds in each row p the final indication V ^P of whether the binary number C ^P of the data set C belongs to the k-minimum set.

Any row p (i.e., V ^P = 1) of the vector V with a value of 1 may indicate that the value of the binary number stored in the same row p of the data set C is qualified as a k-minimum set member. Similarly, all rows p (i. E., V ^P = 0) of a vector V with a value of zero may indicate that the binary number stored in row p of data set C is not eligible to be part of the k-minimum set.

The vector V can be initialized to all zeros at the beginning of the calculation since the k-minimum set is empty. At the end of the calculation, V may contain k qualification indications (i. E., The value of the k bit of the vector V is one and the value of all other bits may be zero). If the bit V ^P of the vector V is set to 1 during the calculation, the associated binary number C ^P of C may be part of the k -minimum set and not stop becoming part of the k -minimum set. The representation of the vector V can only be set. While the k-minimum processor continues to the next column of data set C, the indication may also not be "settled" along the calculation process. (Since a column is treated as an LSB in the MSB, the number defined as the smallest can be larger when the next column is processed without changing its characteristics).

The vector M is a candidate vector and maintains a list of all the rows of the data set C that have a number that can potentially be part of the k-minimum set. The associated binary number of dataset C has not yet been added to the k-min set, but has not yet been excluded from the set, and can potentially also join the set following the procedure of k-min processor 120. As with all other vectors used by the k-minimum processor 120, an indication M ^P of whether or not the binary number C ^P of the data set C can still be considered as a candidate to join the k- It is a q-size vector to hold.

Any row p (i. E., M ^P = 1) of a vector M with a value of 1 may indicate that the value of the binary number stored in row p of data set C may be a candidate to join the k-minimum set. Similarly, all rows p (i. E., M ^P = 0) of a vector M with a value of 0 indicate that the binary number stored in row p of data set C can no longer be considered a candidate to join the k- .

Since the set may not be aligned and the numbers may be randomly spread, the vector M may all be initialized to one since all the numbers of the data set C may potentially be part of the k-minimum set.

If the bit M ^P of the vector M is set to zero during calculation, this indicates that the associated binary number C ^P of C may no longer be considered a potential candidate for the k-minimum set, and the k- While continuing to the next bit for, the indication may not be changed again along the calculation process. Since the number of candidates that can be discontinued is greater than the other binary numbers, it can be permanently excluded from further evaluation.

Vector N is a temporary candidate vector and takes into account the value of the current processed bit of the binary candidate current state and the inverse of which can be stored in vector D according to the past processed bits of C ^P as indicated by vector M. , Keeps a temporary indication NP for each row p that the number C ^P , which is not yet in V, can still be considered as a candidate to join the k-min. N is a logical AND of the vector M and the vector D.

N = M AND D

Any row p (i. E., N ^P = 1) of the vector N with a value of 1 may indicate that the value of the binary number stored in row p of dataset C is still the candidate to join the k-minimum set. Similarly, all rows p (i. E., N ^P = 0) of a vector N with a value of zero may not be considered candidates for a binary number stored in row p of data set C to further join the k- . N ^P would be 1 if and only if the binary number C ^P is not excluded from being a candidate for the previous (ie, M ^P = 1) and the current checked bit of C is 0, ie, D ^P = will be.

Vector T is a temporary member of vectors, whether binary C ^P k- least potentially a member of, or already in, or the minimum set of k- (with the vector display V) candidate (vector N to join the minimum set of k- A temporary indication T ^P for each row p is maintained. T is the logical OR of vector N and vector V.

T = N OR V

Any row p (i.e., T ^P = 1) of the vector T with a value of 1 may indicate that the value of the binary number stored in row p of data set C can be considered as a temporary member of the k- All rows p (i.e., T ^P = 0) of a vector T with a value of zero may indicate that the associated binary number may not be a member of the k-minimum set.

의 역(D = NOT C)이 평가된다 (k개의 최대 값을 찾기 위해서는, C_i가 역의 값 대신에 평가된다). D의 값이 0이면 (즉,

= 1), 수 C^P는 k-최소치 세트에 합류하기에는 너무 커서 잠재적으로 후보 목록 N (N = M 및 D)으로부터 제거될 수 있다. 후보 수가 산출되고 (CNT = COUNT(N OR V)), 원하는 크기인 k-최소치 그룹-k와 비교된다.As described above, the k-minimum processor 120 can work concurrently on all the numbers C ^P stored in the data set C and can iterate over the bits from the MSB to the LSB. It can start with an empty group (V = 0) and assign a candidate state to all binary numbers in the data set (M = 1). At each step of k-minimum processor 120, the bits of column C _i

(D = NOT C) is evaluated (in order to find the k maximum values, C _i is evaluated instead of the inverse value). If the value of D is 0 (that is,

= 1), C ^P may be removed too large hagieneun joining k- minimum value set from the list of candidates potentially N (N = M and D). The number of candidates is calculated (CNT = COUNT (N OR V)) and compared with the desired size k-minimum group -k.

If the CNT (k-min potential set of potential binary numbers) is smaller than required (CNT <k), then all candidates may qualify (since v = N OR V), the search may continue.

If CNT is greater than necessary (CNT > k), all binary numbers having a bit value of 1 in the currently checked bits are removed from the candidate list (M = N), reducing the number of candidates. The remaining candidates may proceed to the next step.

If the CNT meets the required value (CNT = k), then all the candidates can be qualified (V = N or V) and the calculation of the k-minimum processor 120 can be terminated.

6, which is now referenced, is a flow diagram of k-minimum processor 120 functional steps that are constructed and operate in accordance with a preferred embodiment of the present invention. The k-minimum processor 120 functional steps include initialization 610, loop 620, vector computation 630, large set 640, small set 650, and appropriate set 660. The processing steps of the k-minimum processor 120 are also provided below as pseudo-code.

The initialization 610 may initialize the vector V to zero since the k-minimum set may start with the empty set, and the vector M may be initialized to one since all binary numbers in the data set C may be candidates.

Loop 620 may loop over every bit of the binary number of data set C, starting at the MSB and ending at the LSB.

For each processed bit, the vector output 630 may yield the temporary vectors D, N, and T, and the amount of the candidate may be counted. The vector D can be generated as the inverse of the column i and the candidate vector N is generated from the value of the bit i reflected by the vector D holding the inverse of the existing candidate and processed bits (in the vector M). The vector T may be computed as a logical OR between the current member of the set of k-minus reflected by the vector V and the generated candidate vector N. [ The number of candidates in the vector T may be counted as described further below.

If the number of candidates is greater than needed, the large set 640 may update the candidate vector M and continue with the next bit. If the number of candidates is less than required, the small set 650 can add new candidates to member vector V and continue with the next bit, and if the number of candidates is as large as necessary, then the appropriate set 660 will not reach LSB It is possible to update the qualifier marker vector V and exit the loop.

One KMINS (int k, array C)

2 {

3      M: = 1

4      V: = 0

5      FOR i = MSB to i = LSB:

6           D: = not (C [i]);

7           N: = M AND D;

8           T: = N OR V;

9           cnt = COUNT (T);

10           IF cnt> K:

11                M: = N;

12           ELIF cnt <K:

13                V: = T;

14           ELSE:

15                V: = T;

16                EXIT;

17           ENDIF

18      ENDFOR

19 }

Figures 7-11 illustrate the steps of the calculation of the k-minimum processor 120 constructed and operative in accordance with the preferred embodiment of the present invention for the contents of the resulting vector at each step of the exemplary data set and algorithm of Figure 3 Fig. In this example, the required size of the k-minimum set is set to 4 as described above.

7 is an example of the contents of data set C, together with the respective numbers of decimal values to clarify the calculation result and the contents of vector V and vector M after initialization to 0 and 1, respectively.

8 is an example of the state of a different vector after iteration of the k-minimum processor 120 for the MSB with a bit number of 7 in the example of data set C. [ Vector D may contain the inverse value of column 7 of data set C. Next, the vector N can be calculated as a logical AND operation of the vector M and the vector D. Next, the vector T can be calculated as a logical OR operation of the vector N and the vector V, and the number of indications in T is counted. The count value is 5, which is larger than the required k value of 4 in the example. In this case, the vector M is updated to the value of N and the algorithm continues to the next bit. Likewise, Figure 9 is an example of a state of a different vector after iteration of the k-minimum processor 120 for the next bit with a bit number of 6 in the example of data set C. [ As can be seen, the count value in Fig. 9 is 2, which is less than the required value of k = 4. In this case, the vector M is updated to the value of N and the algorithm continues to the next bit.

10 is an example of a different vector after iteration of the k-minimum processor 120 for the next bit with a bit number of five. Vector D may contain the inverse value of column 5 of data set C. [ The vector N can be calculated as a logical AND operation of the vector M and the vector D as before. Next, the vector T can be calculated as a logical OR operation of the vector N and the vector V, and the number of bits having the value "1" is counted. The value of the count is 4, the required set size, so V is updated to the value of T and the algorithm is terminated. At this point the vector V contains a mark (bit value "1") representing a small number of data sets C in all the rows, so that it can be seen that the correct number is indicated by the vector V.

In the exemplary dataset, there are exactly four binary numbers with the smallest value, which can be found by the k-minimum processor 120 after three iterations, although the number of bits of each binary number is eight. It can be seen that the processing complexity is limited by the number of bits of the binary number, not the data set size.

When the binary number is present more than once in the data set, the k-minimum processor 120 may reach the last bit of the binary number of the data set and may not find exactly k items to qualify as a k-minimum member. In this case, an additional set of bits representing the unique index of each binary number of the data set may be used as an additional least significant bit. Since each binary number is associated with a unique index, additional bits can ensure that it generates a unique value for each item in the data set, and can provide the correct amount of items in the k-minimum set.

11, which is now referenced, is an example of an exemplary data set C having an instance of a binary number that is repeated so that the size of the k-min set may be greater than k. (In the example of Fig. 11, the binary number with a decimal value of 56 is repeated twice in row 3 and row 5, and the binary number with a decimal value of 14 is repeated three times in row 8, row 9, and row 10. As a result, there are five items in the k-min set, and k is four). To reduce the number of items in the k-minimum set, the index of each binary number may be processed by the k-minimum processor 120 as the least significant bit of the binary number of data set C. Since the indices are unique, only k indexes will be in the k-minimum set. As shown in Fig. 11, the addition of the index bits produces a k-minimum set with exactly k = 4 members.

As described above, the k-minimum processor 120, constructed and operative in accordance with an embodiment of the present invention, may count the number of indications in the vector, the set bit of the vector T. There are several ways to count the number of set bits in a vector, one of which is to add each number to its immediate neighbor, then two rows of results until the whole vector is counted, four rows of results, etc. And the result is a known pyramid count.

Applicants believe that an efficient count is implemented in the associated memory using the RSP signal as detailed in U. S. Patent Application Serial No. 14 / 594,434, filed January 1, 2015, assigned to the present assignee and assigned to the present assignee. . The RSP signal can be used for an efficient large shift of the bits needed for the display count in the large vector. If the vector is large, a large shift such as shift 16, 256, 2000, etc. may be required to provide an instantaneous shift instead of a single-shift operation.

The RSP is a wired OR circuit capable of generating a signal in response to the positive identification of data candidates in at least one column of the column.

12, which is now referenced, is a schematic diagram of one embodiment that uses an RSP signal to implement an efficient shift for a count operation using the exemplary array 1200. [ Array 1200 may include the following columns: row 1210, vector 1220, location 1230, X-store 1240, RSP signal 1245, and RSP column 1250.

Row 1210 may be the index of a row of array 1200. The array 1200 may be any number of rows such as 32, 64, 128, 256, 512, 1024, 2000, etc., although the array 1200 may have 16 rows. Vector 1220 may be a vector of bits from which the bits from row n are to be relocated to row 0, i. E., To add to the bits in row 0 of another column) Should be copied to position 0. In each row, the value of the bit may be denoted as "y" except for the value stored in row n, which is the value to be shifted and denoted by "X ". All bits of vector 1220 may have a value of "0" or a value of "1 ". The location column 1230 may be a column with a value of "0" in all rows except at row n where the bit (denoted X) will be shifted and the value is set to "1 ". X-Store 1240 may be the result of a Boolean AND operation between vector 1220 and the value of location 1230. [ X-Store 1240 may hold the value X stored in row n of vector 1220 and may null the values of all other rows of vector 1220. [

RSP signal 1245 is the result of an OR operation performed on all cells of X-Store 1240 and may have a value X. [ The value of the OR Boolean operation for all cells in the X-Store 1240 will be the value X, since the value of all bits in the X-Store 1240 is "0" have. The value received at cell RSP signal 1245 may also be written to all cells of RSP 1250, including cell 0, which effectively shifts value X from row n to row 0.

에 의해 정의되고, D^P는 동일한 n개의 속성의 벡터

에 의해 정의된다. m 비트의 이진수 C^P인, 객체 A와 객체 D^P 사이의 거리가 도입된 객체 A와 데이터세트 D의 각각의 객체 D^P 사이에서 산출된다. 거리 C^P는 2개의 0이 아닌 벡터들 사이의 코사인 유사도를 나타낼 수 있다. 본 기술분야에서 공지된 코사인 유사도는 벡터의 각각의 쌍을 벡터의 내적으로 알려져 있는 스칼라 양과 연관시킨다.The k-minimum algorithm described above can be used by a k-nearest neighbor (K-NN) data mining algorithm. In K-NN, D can represent a large data set containing q objects (q is enormously large). D ^P is an object of data set D: D ^P ∈ D, A is the object to be classified. An object is defined by a vector of numerical attributes: A is the vector of n properties

And D ^P is the vector of the same n properties

Lt; / RTI &gt; The distance between the object A and the object D ^P , which is a binary number C ^p of m bits, is calculated between the introduced object A and each object D ^P of the data set D. The distance C ^P can represent the cosine similarity between two non-zero vectors. The cosine similarity known in the art associates each pair of vectors with an internally known scalar quantity of the vector.

Cosine Distance Formula:

. &Lt; / RTI &gt;

The distance C ^P is calculated between object A and each object D ^P in the data set, and stored in binary data in large data set C. The k-minimum algorithm can find k smallest binary numbers in C that represent k nearest neighbors of A within a certain time.

For example, the number of steps required to complete the computation of the k-minimum algorithm for use by the K-NN algorithm is not the number of objects in the data set (q) that can be extremely large, It can be seen that it depends only on the size (the number of bits constituting a binary number representing the distance between A and the object of the data set, that is, m). The calculation of the algorithm can be done simultaneously in all rows of the data set. It will also be appreciated that any addition of objects to the data set may not prolong the processing time of the k-minimum processor 120. If used in an online application, the object take-out time of an object from the data set can remain the same as when the data set grows.

It can be seen that the throughput of a query using the invention described above can be improved by starting the calculation of the next query before the result of the current query is returned to the user. The k-minimum processor 120 may also know that an object may generate an ordered list of items instead of a set by adding numeric indications to each binary number to indicate the repetition identifier that changed the state from candidate to entitlement . Because the smaller binary number is qualified sooner than the larger binary number, the repeat identifier of the smaller binary number may also be smaller than the repeat identifier of the larger binary number of the data set C.

Unless otherwise stated, as is apparent from the foregoing discussion, throughout the specification, the discussion of k minimum numbers applies to the maximum number of k, and vice versa, and may be referred to as the extreme number I understand.

 Applicants have realized that the K-NN process can be used to improve the speed of classifiers and recognition systems in numerous areas such as speech recognition, image and video recognition, referral systems, natural language processing, and the like. Applicants have also realized that K-NN algorithms constructed and operating in accordance with the preferred embodiment of the present invention can be used in fields that were not previously used because they provide good computational complexity of O (1).

13, which illustrates the flow of events of numerous data mining instances where the K-NN algorithm can be used for classification at some point. The system 1300 includes a feature extractor 1320 that extracts features 1330 from an input signal 1310 and a K-NN classifier 1340 that generates a recognition and / or classification 1350 of the items of the input signal 1310. [ ).

Signal 1310 may be an image, voice, document, video, or the like. For images, the feature extractor 1320 may be a Convolution Neural Network (CNN) in the learning phase or the like. For speech, feature 1330 may be a mel-frequency cepstral coefficient (MFCC). In the case of a document, the characteristics include information gain (IG), Chi square (CHI), mutual information (MI), calculated Ng-Goh-Low (NGL) (RS), MSF DF, term frequency for the document frequency (TFDF), and so on. The extracted features may be stored in a device, such as the memory calculation device 100 of FIG. 1, where the K-NN classifier 1340 may operate. Classification 1350 may include classifying a predicted class of items, such as image recognition, or image signals; Ignition detection or noise cancellation for an audio signal; Document classification or spam detection for document signals; And so on.

 For example, the CNN network can know that training can begin using a training set of items for which the classification is known. After a short learning time, the first convergence of the network is observed. The learning phase typically lasts several hours and days for a complete convergence of a stable and reliable network.

According to a preferred embodiment of the invention, learning can be stopped immediately after the start of convergence, and the network can be stored in this "transient" state before complete convergence is achieved.

According to a preferred embodiment of the present invention, the activation value of the training set computed using the network in the "transient" state can be defined as the feature 1330 of each item in the training set, Can be stored together with the classification. It can be seen that the feature can be normalized - that is, the sum of the squares of all activations of each item can be added up to 1.0.

When a new item to be classified is received, the CNN is performed on the item using the network in the transient state, and the K-NN procedure using the stored feature can be used to classify the new item. The K-NN classification of the new item can be performed by calculating the cosine similarity between the feature set of the new object and the item of the database and classifying the new item into the classes of k nearest neighbor classes as described in detail above.

It can be seen that the K-NN algorithm using the k-minimum method described above can replace the last part of the standard CNN.

It can be seen that the addition of the K-NN algorithm can provide a high classification accuracy in a partially trained neural network while drastically reducing the training period time.

Using CNN with a K-NN for classification can replace a fully connected part of the network in applications such as image and video recognition, recommendation systems, natural language processing, and so on.

Applicants have recognized that the KNN process described above may be useful for natural language processing (NLP).

You can consider long texts such as books, written agreements, or the entire Wikipedia. The prior art natural language processor (NLP) creates a neural network that can query these sets of long text queries and get the right answers. To this end, the prior art natural language processor (NLP) uses a recursive neural network (RNN). According to a preferred embodiment of the present invention, the long text may be stored in memory 110 and the KNN process and associated memory array 140 described above may respond to complex queries with a certain computational complexity of O (1) . NLP can also be used for language translation, malware detection, and so on.

The input to the neural network is a key vector and the output is a value vector generated in the neural network by a similarity search between the input key and all other keys in the neural network. To answer a query, for as many iterations as needed until an answer is found, the output can be looped back as the next query. Applicants have found that an associated processing unit (APU), such as the memory computing device 100, can perform any search function, such as a cosine similarity, rather than an exact match, to achieve all that is necessary for natural language processing using a neural network .

End to End Memory Network Architecture - Prior Art

Input Expression: The plot consists of sentences {x _i } from which a set of feature vectors m _i is generated by any other method, such as an RNN, an autocoder, or k-NN already taught. This feature is stored in the neural network. The query q is then also transformed into a feature vector (with the same dimensions as the sentence) using another already-taught embedding. The neural network then computes the similarity as a matrix multiplication of each feature m _i and q. The SoftMax algorithm is then computed to obtain the probability vector. SoftMax can be performed on all neural networks or on k-nearest neighbor vectors.

Output Expression: To generate the output, the probability vector is multiplied by the modified feature vector c _i (which is usually the same as or very similar to feature m _i ). After multiplication, the processor accumulates all N products or just k-nearest neighbors to obtain an output support vector (these results are an intermediate answer to help get the right answer).

Final prediction generation: The intermediate answer is merged with the original query in a final stage (after 3 hops) as a new query for another hop (in a multi-layer variation of the model). The predicted answer is then generated by multiplying the value vector by the associated SoftMax probability and then adding all the vectors to one vector called the "attention vector".

Associative implementation

According to a preferred embodiment of the present invention, the memory computing device 100 is fully scalable and thus has no limit on the size of the text. It can store millions of sentences. A typical associative memory server can have tens of millions of sentences, which is enough to store huge databases. For example, Wikipedia has 2 billion English words. Assuming that they are divided into 500 million sentences, the entire Wikipedia can be stored on 30 to 50 associative memory servers or stored on a single server if using pre-hashing. According to a preferred embodiment of the present invention described in more detail hereinafter, all execution steps occur in parallel in all sentences and have a complexity of O (1).

The memory computing device 100 may be any suitable memory array, such as SRAM, non-volatile, volatile, and non-destructive arrays, and may be any of the memory arrays described in US Pat. No. 9,418,719 -US), and each bitline processor processes one bit of the word, and each word is stored in a column of the associated memory array 140. The bitline processor 114,

Thus, each column of array 140 may have multiple bit line processors. This can be seen in FIG. 14, which shows a portion of the array 140 where six exemplary 2-bit words, A, B, Q, R, X and Y are to be processed. The bits of A1 and B1 may be stored in the bit line processor 114A along the bit line 156 and the bits of A2 and B2 may be stored in the section 114B along the bit line 158. [ Q1 and R1 bits may be stored in bit line processor 114A along bit line 170 and Q2 and R2 bits may be stored in bit line processor 114B along bit line 172. [ The X1 and Y1 bits may be stored in the bit line processor 114A along the bit line 174 and the X2 and Y2 bits may be stored in the bit line processor 114B along the bit line 176. [

Typically, for an M-bit word, there may be M sections each storing different bits of a word. Each section may have a significant number N (e.g., 2048) bit lines, and thus a bit line processor. Each section may provide a row of bit line processors. Thus, N M-bit words may be processed in parallel, and each bit may be processed in parallel by a separate bit line processor.

A typical cell column, such as cell column 150, may store input data to be processed in a first few cells of this column. In Figure 5, the bits of the words A, Q and X are stored in the first cells of the column, while the bits of the words B, R and Y are stored in the second cells of the column. According to a preferred embodiment of the present invention, the remaining cells in each column (which may be 20 to 30 cells in a column) may be left as temporary storage for use during processing operations.

The multiplexer can couple the rows of the bit line processor and the row decoder can activate the appropriate cell in each bit line processor. As described above, the rows of cells in the memory array are connected by word lines, so that the decoder activates the associated word lines of the cell of the bit line processor for reading and, in a different set of bit line processors, Can be activated.

For the natural language processing described above, FIG. 15 illustrates data organization in an associative memory and is now discussed with respect to FIG. There are three main parts 1410-j, one for each of three iterations needed to produce the result. Each section includes three operation sections: a similarity section 1412-j for calculating similarity values for each column, a SoftMax section 1414-j for performing SoftMax calculations on the similarity results, Lt; RTI ID = 0.0 &gt; 1416-j &lt; / RTI &gt; It will be appreciated that the columns of each section are aligned with each other as well as the columns of different iterations. Thus, an operation on the feature x will generally occur within the same column for all operations.

The feature or key vector M ¹ _i of the N input statements is stored in the portion 1412-1 of the memory 110, and each feature vector M ¹ _i is stored in a separate column. Thus, the feature vector M ¹ ₀ is stored in column 0, M ¹ ₁ is stored in column 1, and so on, and each bit of each vector M ¹ _i can be stored in its own bit line processor 114. As discussed above, the feature vector may be the output of a pre-trained neural network or any other vectorized feature extractor and may be a feature of words, sentences, documents, etc., as needed.

The modified feature vector C ¹ _i associated with the N input statements may have the same value of the associated M ^j _i , or some or all of these vectors may be modified in some suitable manner. The modified feature vector C ^j _i may be initially stored in the value section 1416-1. Similar data may be stored in the similarity section 1412-j and the value section 1416-j for the remainder of the repeats j, respectively.

For the similarity section 1410-j, the memory computing device 100 may implement a point matrix multiplication (or cosine similarity) with the input vector q _j for each column in parallel, and the similarity section 1410- The result, which may be the distance between the input vector and the feature vector in each column, may be stored in the associated bit line processor 114 as discussed above. Exemplary matrix multiplication operations are described in U.S. Patent Application No. 15 / 466,889, which is assigned to the common assignee of the present invention and incorporated herein by reference in its entirety. The input vector may be an initial query for iteration 1 and may be a follow on question for the remaining iterations j.

The marker vector T may be used to specify a selected column for forgetting or inserting and updating a new input vector if necessary, and may be implemented in row 1420 and operated for all iterations.

The SoftMax operation described in the Wikipedia entry entitled "SoftMax Function" is based on the result of the point matrix or cosine similarity operation performed on the associated similarity section 1412-j (for the column selected by the marker vector T) j. &lt; / RTI &gt; The SoftMax operation may determine the probability of occurrence for each active column based on the result of the similarity of portion 1412. [ The probability of occurrence has a value of 0 to 1, and the probability is combined to become 1.0.

The SoftMax operation may include a number of exponential operations that may be implemented with a Taylor series approximation and the intermediate data of each operation is stored in the bitline processor of the associated SoftMax portion 1414-j.

In the value section 1416-j, the modified feature vector C ^j _i may each be multiplied by its associated SoftMax value in its bit line processor 114. The first support answer can then be generated by the vector sum of the multiplied C ^j _i vectors. This sum may be accumulated horizontally across all the columns selected by the marker vector T in the Attention Operation. A vector result, weighted by the SoftMax value, that is a weighted sum of the key vectors may be provided to the controller and used to generate a query for the next hop or iteration. 15 shows an initial portion for initial repetition at the bottom of the memory array 110, where additional repetition data is stored in the upper portion thereof. Three iterations are shown, each having an input query q _j and having either a support or final answer as output.

It will be appreciated that the initial query q ₁ may be generated by the query generator using a pre-trained neural network external to the memory computing device 100. The remaining query q _j may be a combination of the original vector query and the attention vector until an answer is reached (usually the third iteration but more iterations are possible).

This combination may be based on an external neural network having two input vectors and one output vector. The input vector is the original query vector q ₁ and the previous iteration's affine vector, and the output is the new vector query. This neural network may be implemented by matrix multiplication on the bit lines of memory or may be implemented externally. It will be appreciated that the initial data stored in the similarity section 1412-j may be the same (i.e., the distance between queries q _j is about the same data). Likewise, the initial value data stored in the value section 1416-j may be the same (i.e., the data to be multiplied by the SoftMax value are the same).

Performance

When all sentence features are stored in memory, the matrix multiplication requires the size of the query vector multiplied by 100 cycles. Assuming less than 10 features per sentence, obtain 1000 clocks in parallel for every N sentences, or 1 μsec (using 1 GHz clock) for every N. SoftMax requires approximately 1μsec, and multiplication and accumulation requires 4μsec. Three hops / repeats require 3X (1 + 1 + 4) to 20μsec and enable 50,000 queries per second.

16, the alternative system 1500 shown in FIG. 16 may include an association memory 1510 that may be large enough to handle only one iteration and other elements that deal with the remaining calculations have.

As in the previous embodiment, associative memory 1510 includes a similarity section 1512 for computing a feature vector referred to herein as a " key ", a SoftMax section 1514 for implementing a SoftMax operation, And a value section 1516 for operating on the &lt; / RTI &gt; This embodiment may execute all the hops in memory 1510 within a certain time. As shown in FIG. 16, some operations take place in memory 1510, while other operations take place outside memory 1510. Performance is approximately the same as the short-to-end implementation, approximately 6 microseconds per hop.

Flexibility to any long memory network

Various types of memory networks, such as the Directly Reading Documents (Miller, &lt; RTI ID = 0.0 &gt; Jason, et al., EMNLP 2016). &Lt; / RTI &gt;

While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

Claims (19)

As a system for natural language processing,
A memory array having rows and columns, the memory array comprising: a similarity section for initially storing a plurality of keys or feature vectors; a SoftMax section for determining a probability of occurrence of the keys or feature vectors; A value section for initially storing a vector, and a marker section, and wherein an operation on one or more columns of the memory array is associated with a feature vector to be processed; And
An in-memory processor
Wherein in the in-memory processor, in each column indicated by the marker section:
Computing a similarity in the similarity section between each of the feature vectors and the vector query stored in each of the displayed columns;
A SoftMax operation in the SoftMax section to determine an associated SoftMax probability value for each of the indicated feature vectors;
Multiplying the associated SoftMax probability value by each modified feature vector stored in each column of the indicated column; And
The vector sum-vector sum in the value section, which accumulates an attention vector sum of the output of the multiplication operation, may be used to generate a new vector query for further iteration or to generate an output value in the final iteration -
Wherein the memory array is activated in parallel. The method according to claim 1,
Wherein the memory array comprises a plurality of arithmetic units, one for each iteration of a natural language processing operation, and each arithmetic unit is divided into the sections. The method according to claim 1,
Wherein the memory array is one of SRAM, non-volatile, volatile, and non-destructive arrays. The method according to claim 1,
Wherein the memory array includes a plurality of bit line processors, one for each column of each of the sections, each bit line processor operating on one bit of data of an associated section. The method according to claim 1,
Further comprising a neural network feature extractor for generating the feature vector and the modified feature vector. The method according to claim 1,
Wherein the feature vector comprises a feature of a word, a sentence, or a document. The method according to claim 1,
Wherein the feature vector is an output of a pre-trained neural network. The method according to claim 1,
Further comprising a pre-trained neural network for generating an initial vector query. 9. The method of claim 8,
Further comprising a query generator for generating an additional query from the initial vector query and the attachment vector sum. 10. The method of claim 9,
Wherein the query generator is a neural network. 10. The method of claim 9,
Wherein the query generator is implemented as a matrix multiplier on a bit line of the memory array. As a method for natural language processing,
Comprising: a memory array having rows and columns, the memory array comprising: a similarity section for initially storing a plurality of keys or feature vectors; a SoftMax section for determining a probability of occurrence of the keys or feature vectors; A value section for initially storing a feature vector, and a marker section, and wherein an operation on one or more columns of the memory array is associated with a feature vector to be processed; And
In each column denoted by the marker section:
Computing a similarity in the similarity section between each of the feature vectors and the vector query stored in each of the displayed columns;
A SoftMax operation in the SoftMax section to determine an associated SoftMax probability value for each of the indicated feature vectors;
Multiplying the associated SoftMax probability value by each modified feature vector stored in each column of the indicated column; And
The vector sum-vector sum in the value section that accumulates the sum of the attack vector of the output of the multiplication operation will be used to generate a new vector query for further iteration or to generate an output value in the final iteration -
The step of activating the memory array to perform in parallel
&Lt; / RTI &gt; 13. The method of claim 12,
Wherein the memory array comprises a plurality of bit line processors, one for each column of each of the sections, the method further comprising each bit line processor operating on one bit of data of an associated section. Methods for language processing. 13. The method of claim 12,
Further comprising generating the feature vectors and the modified feature vectors using a neural network and storing them in the similarity section and value section, respectively. 13. The method of claim 12,
Wherein the feature vector comprises a feature of a word, a sentence, or a document. 13. The method of claim 12,
Further comprising generating an initial vector query using a pre-trained neural network. 17. The method of claim 16,
Further comprising generating an additional query from the initial vector query and the affine vector sum. 18. The method of claim 17,
A method for natural language processing that utilizes a neural network to generate additional queries. 18. The method of claim 17,
Wherein generating the additional query comprises performing a matrix multiplication on a bit line of the memory array.

Images