US20200285948A1

US20200285948A1 - Robust auto-associative memory with recurrent neural network

Info

Publication number: US20200285948A1
Application number: US16/646,071
Authority: US
Inventors: James K. Baker
Original assignee: D5AI LLC
Current assignee: D5AI LLC
Priority date: 2017-09-28
Filing date: 2018-09-19
Publication date: 2020-09-10
Also published as: US20200265320A1; US11354578B2; US20190095798A1; WO2019067248A1; EP3688678A4; US11410050B2; EP3688676A1; US20200320371A1; US20220383131A1; US20200285939A1; US11531900B2; US11461661B2; US20200279188A1; US20220335305A1; WO2019067236A1; EP3688677A1; US11074505B2; EP3688676A4; EP3688678A1; CN111226236A

Abstract

Computer systems and computer-implemented methods recursively train a content-addressable auto-associative memory such that: (i) the content addressable auto-associative memory system is trained to produce an output pattern for each of the input examples; and (ii) a quantity of the learned parameters for the content-addressable auto-associative memory is equal to the number of input variables times a quantity that is independent of the number of input variables. The quantity of learned parameters for the content-addressable auto-associative memory system can be varied based on the number of input examples to be learned.

Description

PRIORITY CLAIM

The present application claims priority to U.S. provisional patent application Ser. No. 62/564,754, entitled “Aggressive Development with Cooperative Generators,” filed Sep. 28, 2017, which is incorporated herein by reference in its entirety.

BACKGROUND

In information retrieval and in data science it is often necessary to retrieve an item from a large collection of data without knowing where the data item is stored. Often there is only an incomplete, partial description of the item being sought. The data collection may contain billions or even trillions of items, so an exhaustive search of the collection may be prohibitively expensive and time consuming. In some cases, it is not known whether an exact match of a desired item is in the collection. In some cases, it is merely necessary to find any item that matches a query to some degree.
A memory system that can retrieve a data item based on a description of its content rather than by knowing its location is called a “content addressable” memory. If, furthermore, the retrieval can be done in spite of partial or imperfect knowledge of the contents, the memory system is said to be a “robust content addressable” memory. For example, a Hopfield network may be used as a robust content addressable memory. In addition, such a Hopfield network is auto-associative. A variation on a Hopfield network, called a “Bi-directional associative memory (BAM)” is hetero-associative. However, the number of learned parameters (a measure of memory capacity) in a Hopfield network is just the number of undirected arcs, which is to say, the number of unordered pairs of input variables. That is, the number of learned parameters of a Hopfield network is roughly one-half the square of the number of input variables. For a BAM, the number of learned parameters is the product of the number of input variables times the number of output variables.
There is no flexibility in the number of learned parameters for these networks. In either case, the capacity of the memory is determined by the number of input and output variables rather than by the number of data items to be learned. In an auto-associative memory, the number of scalar values to be represented is the product of the number of data items times the number of input variables. Thus, with say, 100 input variables, a Hopfield network does not have the capacity to learn a data base with say, 100,000,000 data items. On the other hand, a high-resolution color image may have twenty-million pixels in three colors, so the number of learned parameters in a Hopfield network would be on the order of two hundred trillion for a black-and-white image or sixteen quadrillion for the full color image, which would be totally impractical.
In addition, Hopfield networks and BANIs are connected via undirected arcs and, unlike deep layered neural networks, do not have hidden layers and are not trained by the highly successful back propagation computation commonly used in deep learning.

SUMMARY

The present invention, in one general aspect, uses a machine learning system, such as a deep neural network, to implement a robust, content-addressable auto-associative memory system. The memory is content-addressable in the sense that an item in the memory system may be retrieved by a description or example rather than by knowing the address of the item in memory. The memory system is robust in that an item can be retrieved from a query that is a transformed, distorted, noisy version of the item to be retrieved. An item may also be retrieved based on an example of only a small portion of the item. The associative memory system may also be trained to be a classifier. The memory system is recurrent and auto-associative because it operates by feeding its output back to its input.
The memory system may be based on a layered deep neural network with an arbitrary number of hidden layers. The number of learned parameters may be varied based on the number of data items to be learned or other considerations. Embodiments based on such deep neural networks may be trained by well-known deep learning techniques such as stochastic gradient descent based on back propagation.
These and other benefits realizable with the present invention will be apparent from the description below.

FIGURES

Various embodiments of the present invention are described herein by way of example in conjunction with the following figures.

FIG. 1 is a block diagram of an illustrative embodiment of the invention described herein.

FIG. 1A is a block diagram of another aspect of an illustrative embodiment of the invention.

FIG. 1B is a flow chart of a process for training the associative memory according to various embodiments of the present invention.

FIGS. 2A and 2B are flow charts of aspects of an illustrative process for training various embodiments of the invention.

FIG. 3 is a diagram of a computer system for implementing various embodiments of the invention.

FIG. 4 is a diagram of a type of neural network that may be used in various embodiments of the invention.

FIG. 5 is a diagram depicting unrolling of a recurrent network.

DESCRIPTION

FIG. 1 is an illustrative embodiment of a robust auto-associative memory system 100 with corrective training. The associative memory unit 104 is a large machine learning system, for example a deep neural network, such as the example deep neural network illustrated in FIG. 4. The machine learning system 104, by itself, may be a deep feed-forward neural network. It becomes recurrent and auto-associative because of the feedback 99 from the output prediction of the full pattern (105) back to the input to machine learning system 104. Because the machine learning system 104 may, for example, be a layered deep, feed-forward neural network, it may have an arbitrary number of learned parameters. The learned parameters may comprise trainable parameters for the machine learning system 104, such as the weights of directed arcs in a deep, feed-forward neural network and/or activation biases for nodes in the network. In an auto-associative memory, the number of learned parameters needs to be the number of data items times the number of variables in each data item, perhaps times a small multiple (e.g., a number greater than but close to 1.0) to allow for redundancy and robustness. The number of learned parameters in the machine learning system 104 may be adjusted to meet this requirement. In particular, the number of learned parameters may be based in part on the number of data items to be learned (times the number of input variables). The factor that is based on the number of data items may be either much smaller than the number of input or output variables or may be much larger than the number of input or output variables. Furthermore, extra learned parameters may be used to achieve better performance. In addition, the machine learning system 104 may have any number of layers enabling it to compute arbitrarily complex functions, for example to better model and compensate for transformations and degradations to the input. If the machine learning system 104 is a deep neural network, the recurrence due to the feedback from the output 105 back to the input of network 104 may be unrolled by making multiple copies of the network 104, producing a single, large feed-forward neural network. This feed-forward deep neural network has directed arcs rather than undirected arcs as in a Hopfield network or in a bi-directional associative memory (BAM). Thus, it may be trained by stochastic gradient descent computed by feed-forward activation and back propagation of the partial derivatives of the objective. Deep neural networks and the methods of stochastic gradient descent, unrolling the recurrence, feed-forward activation, and back propagation of partial derivatives are discussed in association with FIG. 4. Embodiments of the machine learning system 104 other than deep neural networks may also have as many learned parameters as necessary, including having the number of learned parameters be based in part on the number of data items to be learned.
The task of an auto-associative memory is to memorize its training data. A robust auto-associative memory 100 not only memorizes its training data, it is able to retrieve an example from its training data given only a partial input or a degraded input. In FIG. 1, the dashed arrows from blocks 106, 107 to block 105 represent backpropagation for training the associative memory 104 and the solid arrows represent feedforward activation, with backpropagation in the opposite direction from feedforward activation. If the associative memory unit 104 is not a deep neural network, a similar training process may be used if the machine learning system implementing the associative memory 104 can be trained to avoid producing negative examples as well as to produce positive examples.
An illustrative example of a computer system 300 that may perform the computations associated with FIG. 1 and other figures is shown in FIG. 3.
FIG. 1 shows a system diagram of a robust auto-associative memory machine learning system 100 configured for an illustrative embodiment of an aspect of the invention in which a computer system 300 performs preliminary training of a robust associative memory 104, with a moderate level of transformations and data augmentations of input 101 optionally performed by the computer system 300 in step 102. A training process with more extreme transformations and data augmentations is discussed in association with an illustrative flow chart in FIGS. 2A and 2B. In either the preliminary training process illustrated in FIG. 1 or the process illustrated in FIGS. 2A and 2B, the number of learned parameters in the associative memory unit may be increased as necessary to achieve the required performance. For example, the number of learned parameters may be varied proportional to the number of data items to be learned. If the associative memory unit 104 is a deep neural network, the number of nodes in a layer may be increased or the number of layers may be increased. An illustrative example of an associative memory machine learning system during operational use is shown in FIG. 1A.
In the illustrative embodiment of FIG. 1, the computer system 300 presents each training example (101) multiple times to the associative memory 104, each time with zero or more randomly generated transformations, degradations, and subsampling (102). As indicated by the dotted arrow from 101 to 104, in some embodiments, there is no transformation done in step 102. In some embodiments, the original input data pattern (101) may be transformed (102) by translation, rotation, or any linear transformation. In some embodiments, the original data (101) may be degraded (102) by additive or multiplicative noise. In some embodiments, the input (101) may be changed in other ways that are used for data augmentation (102) in training classifiers. Data augmentation is well-known to those skilled in the art of training machine learning systems. In some embodiments, only a subset of the original input variables (101) or only a subset of the transformed input variables (102) are retained for the next transformation or to be sent to the associative memory 104. In some embodiments, each of these types of transformations may be used in combination with others, with each types of transformation possibly used multiple times, as indicated by the arrow from block 102 back to itself.
In some embodiments, the amount of a translation, rotation, or other transformation at step 102 may be characterized by a parameter, such as the distance of the translation or the angle of the rotation. This transformation-characterizing parameter may directly be controlled as a hyperparameter or its maximum magnitude may be controlled by a hyperparameter. Hyperparameter values may be set by the system designer or may be controlled by a second machine learning system, called a “learning coach.” A learning coach may also adjust the number of learned parameters based on the number of data items to be learned, such as, for example, by adding nodes and/or layers, to thereby increase the number of arcs that need weights and the number of nodes that need biases.
A learning coach is a second machine learning system that is trained to help manage the learning process of a first machine learning system. Learning coaches are described in more detail in the following applications, which are incorporated herein by reference in their entirety: published PCT application Pub. No. WO 2018/063840 A1, published Apr. 5, 2018, entitled “LEARNING COACH FOR MACHINE LEARNING SYSTEM”; and PCT Application No. PCT/US18/20887, filed Mar. 5, 2018, entitled “LEARNING COACH FOR MACHINE LEARNING SYSTEM.”
In the embodiment illustrated in FIG. 1, the computer system 300 trains the robust auto-associative memory machine learning system 100 to predict the full original pattern 105 with the original, untransformed input data example as its target 106. The computer system 300 also feeds back (via line 99) predicted pattern 105 as input to the associative memory 104. That is, the associative memory unit 104 is a recursive system, with its output repeatedly fed back to its input. The objective of the recursion is to produce a better match between the predicted pattern 105 and the target 106 with each round of the recursion. With each round of feedback, the computer system 300 causes the associative memory unit 104 to refine its prediction 105 of the full, undegraded pattern 106. In this recursion, the computer system 300 may cause the associative memory 104, for example, to recover some of the missing parts of the target 106 and to remove part of the noise and distortion from the pattern 105 in the first round of the recursion. With that more complete, somewhat cleaner, input, the computer system 300 may then cause the associative memory unit 104 to recover more of the original pattern 106 in the next round, and so on, until a stopping point is met, i.e., the output of the associative memory 104 satisfactorily matches the original pattern. The memorized patterns are the fixed points of this recursive process. In some embodiments, examples of the input 101 and of the prediction 105 for later rounds of the recursion are saved by the computer system 300 in storage 109. The choice of when a pattern should be saved may be made by fixed rules set by the system developer or may be made by a learning coach.
In that connection, FIG. 1B is a flow chart illustrating the training process of the associative memory system 100 according to various embodiments. At step 120, the computer system 300 selects the first epoch of training data. At step 121, the computer system 300 selects one of the training examples from the original epoch and transforms the selected training example at step 122. As described above in connection with block 102 of FIG. 1, the transformation at step 122 may comprise a transformation, augmentation, sub sampling, etc. of the selected training example. At step 123, the computer system 300 trains the associative memory unit 104 with the transformed training example. Step 123 may be repeated recursively as described above, as indicated by the feedback loop from step 124, until the stopping point is met (e.g., the output of the associate memory unit 104 satisfactorily matches the training example). Once the stopping point is met for training the associative memory for the transforming training example, the process advances to step 125. At step 125, if the selected training example is to be transformed in a different way, the process returns to step 122, where the training example selected at step 121 is transformed in a second (or additional) random manner, and then the associative memory 104 is recursively trained at steps 123-124 with the training example transformed in a second (or additional manner). This second, outer loop (i.e., steps 122 to 125) can be repeated for a desired number of transformations to a training example. That number may be a fixed number (e.g., set by the system designer) or controller by a hyperparameter. Step 125 is also optional. That is, in other embodiments, a training item selected at step 121 is only transformed once for training the associative memory.
If there are no more transformations to be made to the selected training example at step 125, or if step 125 is omitted, the process advances to step 126, where, as shown by the feedback loop from step 126 back to step 121, the process is repeated for the next training example in the epoch, and so on until the process has been performed for all of the training examples in the epoch. The process is then repeated for multiple epochs until the training has converged or another stopping criterion, such as a specified number of epochs, is met, as indicated by the feedback loop from the decision step 127 back to step 120.
The use of the fixed points of the recursive process to represent memorized data items results in the robustness and other remarkable properties for the robust auto-associative memory system 100. For example, with the recursive process, an entire complex image may be recovered from a small piece of the image if the small piece occurs in only one image in the set of images memorized by the robust auto-associative memory unit 100. Similarly, a memorized document (e.g., word processing document, pdf file, spreadsheet, presentation, etc.) may be recovered from a small, unique portion of text; or an audio recording may be recovered from a small interval of sound. As already mentioned, the auto-associative memory system 100 may be trained to be robust against a wide variety of transformations or noise. As another example, a work of art or a photograph may be retrieved from a memorized database from a sketch-like query. This robustness and other properties are further enhanced by another training process for which an illustrative embodiment is discussed in association with FIGS. 2A and 2B.
The training process illustrated in FIGS. 1 and 1B also may include negative feedback from negative examples, such as 107. For example, the recursive function implemented by the associative memory 104 may have other fixed points in addition to the memorized training data examples 101. In some embodiments, the computer system 300 may train the associative memory unit 104 to eliminate such extra fixed points by training it not to generate them as output by negative feedback from the undesired fixed points as negative examples 107. Another example of use of negative feedback is for the computer system to cause the auto-associative memory unit 104 to forget or erase the memory of a pattern it has been previously trained to remember. Another example, in some embodiments of this invention, the computer system 300 can train a separate associative memory (not shown) to learn all the patterns in one classification category but not to learn examples from any other category. The negative examples then give negative feedback to the separate associative memory when it outputs a pattern that matches a different category than the intended category, for example, as judged by an independent classifier (not shown). Other uses of negative examples are discussed in association with FIGS. 1A, 2A and 2B.
Normal feedback may be represented by a loss function that has its minimum at a data item that is an intended target or positive example. Negative feedback may be represented by a loss function that has its maximum at a data item that is a negative example. For either type of feedback, the partial derivatives of the loss function may be computed by computer system 300 using back propagation if the associative memory 104 is a neural network. Back propagation is well-known to those skilled in the art of training neural networks and is discussed in association with FIG. 4. If the associative memory unit 104 is some other type of machine learning system that supports training on negative examples, then preferably computer system 300 may train the associative memory unit 104 on negative examples 107 by whatever method is appropriate for the type of machine learning system being used in the associative memory 104.
In some embodiments, the number of learned parameters for the associative memory 104 may be adjusted during the learning process. For example, such a change may be based on testing the performance of the associative memory 104 on new data not used in the training. The number of learned parameters may be increased to increase the capacity of the memory 104 or the number of learned parameters may be decreased if testing reveals that there is spare capacity. Where, for example, the associative memory 104 comprises a deep neural network, additional learned parameters may be added by adding additional directed arcs, by adding additional nodes, or by adding additional layers to the network.
FIG. 1A is a system diagram of the robust auto-associative memory machine learning system 100 of FIG. 1 in operational use. FIG. 1A is an illustration of the same system as illustrated in FIG. 1, except configured for operational use rather than for training. In particular, components 104, 105, and 109 in FIG. 1A are the same as components 104, 105, and 109, respectively, in FIG. 1. However, as shown in FIG. 1A, during operational use, the computer system 300 provides any input 103 to the associative memory 104, not just training data items 101 and their transformations 102, since in operational use, the purpose of the associative memory is to output a pattern 105 that corresponds to a pattern from its training examples 101 (see FIG. 1) in response to any input 103.
In the embodiment illustrated in FIG. 1A, the computer system 300 provides the input 103 directly to the associative memory 104. That is, there is no distortion, etc., as in block 102 of FIG. 1. Having presented the associative memory 104 with an input data item 103, the computer system 300 computes the output of associative memory 104. For example, if associative memory is implemented as a deep neural network, the computer system 300 does a feed-forward activation computation from the input 103 to compute the activations of the nodes in the network 104 and then the activations of the output nodes 105. The feed-forward activation computation is explained in association with FIG. 4 and is well-known to those skilled in the art of deep neural networks.
Once computer system 300 has computed the output 105 of the associative memory 104, the computer system 300 applies the output 105 recursively as input to the associative memory 104. The computer system 300 repeats this recursion until the recursion converges or a stopping condition is met. A possible stopping condition is detection of an infinite cycle. An infinite cycle may be detected by observing that an output for a cycle is identical to the output for some previous cycle. In some embodiments, the computer system 300 may save an output example 109 at convergence or at any stage of the recursive process, including the input 103, for use in later training or for detection of an infinite cycle. In some embodiments, all stages of the recursion process are saved. In some embodiments, examples to be saved may be selected by a learning coach.
Although the associative memory 104 is a machine learning system, such as a deep neural network, its task is not to classify its input as in most machine learning tasks. Instead, as its name implies, the associative memory 104 has the task of retrieving from its “memory” of the training data 101 (see FIG. 1) a training data item 105 that is “associated” with input 103. The set of input items 103 that may be associated with a specific training data item 105 is the set of input items 103 such that the recursion converges to the specific training data item.
If the recursion converges to an output that is not a training data item, in some embodiments the computer system 300 may save the output at convergence in 109 for later use in training as a negative example 107 of FIG. 1. In some embodiments, the computer system 300 may save some or all of the data items that occur in an infinite cycle as negative examples.
If the recursion converges to an output that is a training data example, the computer system 300 may save the input and one or more of the intermediate stage outputs in association with the training data example as positive examples 108 for future training.
For either negative or positive examples transferred by computer system 300 to block 108, if the learned parameters of the associative memory 104 are being trained during operation, such as with adaptive training, then the computer system also saves in 108 a snapshot of the current values of the learned parameters of the associative memory unit 104 and links the snapshot to the corresponding positive or negative example.
A negative example saved in block 108 of FIG. 1A is available to use as a negative example 107 in FIG. 1.
FIGS. 2A and 2B together represent a flow chart of an illustrative embodiment of another training process. The training process illustrated in FIGS. 2A and 2B may be based on a system such as the illustrative system 100 shown in FIG. 1, but the training process illustrated in FIGS. 2A and 2B differs from the training process described above in association with FIG. 1 in several important aspects. For example, in the illustrative embodiment of FIGS. 2A and 2B, the training process may make use of classification category labels and the robust auto-associative memory system 100 of FIG. 1 may be used as a classifier. In another aspect, the training process illustrated in FIGS. 2A and 2B is an iterative process that actively increases the amount or degree of the transformations 102 applied to the input 101 and tests to see if the amount or degree of a transformation 102 is too great.
FIG. 2B illustrates the iterative aspect of the training process. FIG. 2A illustrates some aspects of the initial part of the training process and some aspects that apply to the training process as a whole rather than to individual steps in the process.
In step 200 of FIG. 2A, the computer system 300 obtains a preliminary auto-associative memory system, such as the system 100 illustrated in FIG. 1. The computer system 300 may obtain the preliminary auto-associative memory system by using the training process discussed in association with FIGS. 1 and 1B, or may obtain the preliminary auto-associative memory from a previous use of the training process illustrated in FIGS. 2A and 2B. Preferably, the previous training of the preliminary auto-associative memory system 100 is done using the same training data set as to be used in the current training, or a subset of the training data set to be used in the current training. In some embodiments, the preliminary auto-associative memory system 100 may be trained to “forget” a previous training data example by using that previous example as a negative training example. In some embodiments, the preliminary auto-associative memory system 100 may be untrained; for example, it may be a deep neural network with random initialization of the connection weights. Such random initialization is well-known to those skilled in the art of training deep neural networks.
In step 201, the computer system 300 obtains training data that is labeled with classification category labels. The computer system 300 may store the training data in memory. In step 202, the computer system 300 sets aside some of the training data obtained at step 201. The data set aside at step 202 may be used for development testing by a learning coach.
The computer system 300 applies rules and procedures 203, 204, and 205 throughout the training process, as described below. In some embodiments, in procedure 203, the computer system 300 uses the auto-associative memory system 100 (e.g., the one obtained at step 200) as a classifier. As such, at least some of the training data items obtained at step 201 are labeled with a classification category. In operation, the illustrative embodiment of the auto-associative memory system 100 shown in FIG. 1A accepts any input 103 and attempts to compute an output pattern 105 that closely matches a training data item. In step 201, the computer system has obtained training data with classification category labels. In some embodiments of procedure 203, the computer system 300 includes the classification category or an identifier associated with the classification category as part of the input data item to the auto-associative memory unit 104 and retrieves that information as part of the output pattern 105. The identifier associated with a training data item may be a hash code or other index with check sums or error-correcting encoding. The computer system uses this redundant encoding to determine its estimate of the index of the actual item or of the classification category of the untransformed input 101. Thus, in this illustrative embodiment, the auto-associative memory system 100 acts as a classifier.
In procedure 204, the computer system 300 may control the maximum amount or degree to be allowed of a transformation or data augmentation in step 102 of FIG. 1. The maximum amount allowed may be represented by a hyperparameter “delta.” In some embodiments, the maximum magnitude of a translation may be a specified function of delta; the maximum degree of rotation may be some other specified function of delta; the maximum standard deviation of noise may be yet another specified function of delta; the maximum fraction of input variables that may be deleted may be a specified function of delta; and/or the maximum degree of each other transformation may be a specified function delta. In some embodiments, these specified functions may be set by the system developer. In some embodiments, a learning coach may be trained to optimize these specified functions. The hyperparameter delta as implemented in this illustrative embodiment of procedure 203 may be used in step 211 of the iterative training loop shown in FIG. 2B.
In operation of the auto-associative memory system 100 in the embodiment illustrated in FIG. 1A, at convergence the output pattern 105 may fail to be any of the training data items. In this case, under procedure 205 in FIG. 2A, the computer system 300 may train the auto-associative memory 100 using the unlabeled converged output pattern 105 as a negative example.
After steps 200, 201, and 202 and putting in place the procedures 203, 204, and 205 for operation during the training process, the computer system proceeds to the iterative training loop illustrated in FIG. 2B, beginning at step 211. In step 211, the computer system 300 trains the auto-associative memory 100 with a larger value of delta than has been used in previous passes through the loop from step 211 to step 217, if there have been any previous passes. In some embodiments, the number of learned parameters may also be adjusted in step 211. The number of learned parameters may be either increased or decreased, as indicated by the development testing in steps 212 and 216.
In the first pass through the loop from step 211 to step 217, preferably delta is set at a small value. In some embodiments, in the first pass the value of delta is set to zero, representing that no transformations, data augmentations, or input variable deletions are to be performed in step 102 of FIG. 1. In step 211 during later passes through the loop from step 211 to step 217, the computer system 300 increases the value of delta by an amount controlled by a hyperparameter.
In step 212, the computer system 300 measures the performance of the auto-associative memory system 100 acting as a classifier. Preferably, this performance measurement is made on development data that has been set aside from the training data, as specified in step 202 of FIG. 2A.
In step 213, the computer system 300 compares the performance measured in step 212 with measures of performance from previous passes through the loop from step 211 to step 217 or from the preliminary auto-associative memory obtained in step 200 of FIG. 2A. If there is degradation by more than a specified amount, the computer system 300 proceeds to step 214, otherwise the computer system returns to step 211. In some embodiments, the computer system 300 allows no degradation in performance in step 213. In other words, if there is possible further improvement (i.e., more than zero degradation), the process returns to step 211; otherwise, the process advances to step 214. In other embodiments, some degradation in performance is allowed but the limit on the amount of degradation in performance is measured against the best previous performance, not just against the performance of the immediately preceding pass through the loop from step 211 to step 217.
In step 214, the computer system generates data according to step 102 of FIG. 1 with various types of transformations, data augmentations, noise, and input variable sub sampling at or near the limiting value delta. In step 214, the computer may also select data that was generated in step 211. In other words, in step 214, the computer system is generating or selecting transformed input examples that would not have been generated in the previous passes through the loop from step 211 to step 217 with smaller values of delta. Thus, these data examples would not have been generated in the previous iteration that had better performance than the performance measured in step 212 of the current pass through the loop. Some of the data examples generated or selected in step 214 in the current round may be responsible for the degradation in measured performance. In step 214, the computer system is attempting to replicate on training data the kind of degradation in performance that was measured on development data in step 212.
In step 215, the computer system classifies data generated in step 214 and counts as a misclassification any data 102 that is classified with a category different from the category of the untransformed data 101. Under control of hyperparameters, some or all of these misclassifications are used for training as negative examples.
In step 216, the computer system again measures the performance of the auto-associative memory system 100 acting as a classifier with the performance measured on set-aside development data. In some embodiments the same set of set-aside development data may be used for performance comparisons in step 213 so that the performance difference does not depend on differences in the data on which the performance is measured. In some embodiments, a second set of set-aside development data is used to confirm the cumulative progress of multiple passes through the loop from step 211 to step 217.
In step 217, the computer system again compares the performance of the current system with the previous performance. If there has been an improvement in performance, the computer system returns to step 211 to continue the process with a larger value of delta. If there has been no improvement in performance, the computer system backs up the learned parameter values to the best performance values previously obtained and terminates the training process.
In some embodiments there may be other remedial actions to reduce the classification errors caused when delta is increased. In such embodiments, the computer system may experimentally try some of these other remedial actions and return to step 214 multiple times to try to find an improvement and proceeding to step 211 before eventually deciding to proceed to step 218. For example, in some embodiments, a learning coach may modify the specified function of delta for one or more types of transformation eliminating or limiting the amount of increase in such a transformation. In some embodiments, the computer system may merely return to step 214 multiple times to generate more negative examples.
Although the auto-associative memory unit 104 has generally been described as a single neural network, such as shown in FIG. 4, many variations are possible within the scope and spirit of the invention. As already mentioned, other types of machine learning systems may be used if they support back propagation of partial derivatives of the objective or if they merely support training against negative examples by some other means. As another example, the auto-associative memory unit 104 may be an ensemble of neural networks or other machine learning systems. As a particular example, with labeled training data, a separate auto-associative memory unit may be trained for each classification category with data exclusively consisting of the single classification category.
FIG. 3 is a diagram of a computer system 300 that could be used to implement the embodiments described above, such as the process described in FIG. 1. The illustrated computer system 300 comprises multiple processor units 302A-B that each comprises, in the illustrated embodiment, multiple (N) sets of processor cores 304A-N. Each processor unit 302A-B may comprise on-board memory (ROM or RAM) (not shown) and off-board memory 306A-B. The on-board memory may comprise primary, volatile and/or non-volatile, storage (e.g., storage directly accessible by the processor cores 304A-N). The off-board memory 306A-B may comprise secondary, non-volatile storage (e.g., storage that is not directly accessible by the processor cores 304A-N), such as ROM, MDs, SSD, flash, etc. The processor cores 304A-N may be CPU cores, GPU cores and/or AI accelerator cores. GPU cores operate in parallel (e.g., a general-purpose GPU (GPGPU) pipeline) and, hence, can typically process data more efficiently that a collection of CPU cores, but all the cores of a GPU execute the same code at one time. AI accelerators are a class of microprocessor designed to accelerate artificial neural networks. They typically are employed as a co-processor in a device with a host CPU 310 as well. An AI accelerator typically has tens of thousands of matrix multiplier units that operate at lower precision than a CPU core, such as 8-bit precision in an AI accelerator versus 64-bit precision in a CPU core.
In various embodiments, the different processor cores 304 may train and/or implement different networks or subnetworks or components. For example, in one embodiment, the cores of the first processor unit 302A may implement the auto-associative memory 104 and the second processor unit 302B may implement the learning coach. For example, the cores of the first processor unit 302A may train the machine learning system (e.g., neural network) of the auto-associative memory 104 according to techniques described herein, whereas the cores of the second processor unit 302B may learn, from implementation of the learning coach, the hyperparameters for the auto-associative memory 104. Further, where the associate memory 104 comprises an ensemble of machine learning systems, different sets of cores in the first processor unit 302A may be responsible for different ensemble members. One or more host processors 310 may coordinate and control the processor units 302A-B.
In other embodiments, the system 300 could be implemented with one processor unit 302. In embodiments where there are multiple processor units, the processor units could be co-located or distributed. For example, the processor units 302 may be interconnected by data networks, such as a LAN, WAN, the Internet, etc., using suitable wired and/or wireless data communication links. Data may be shared between the various processing units 302 using suitable data links, such as data buses (preferably high-speed data buses) or network links (e.g., Ethernet).
The software for the various compute systems described herein and other computer functions described herein may be implemented in computer software using any suitable computer programming language such as .NET, C, C++, Python, and using conventional, functional, or object-oriented techniques. Programming languages for computer software and other computer-implemented instructions may be translated into machine language by a compiler or an assembler before execution and/or may be translated directly at run time by an interpreter. Examples of assembly languages include ARM, MIPS, and x86; examples of high level languages include Ada, BASIC, C, C++, C#, COBOL, Fortran, Java, Lisp, Pascal, Object Pascal, Haskell, ML; and examples of scripting languages include Bourne script, JavaScript, Python, Ruby, Lua, PHP, and Perl.
FIG. 4 is a drawing of an example of a type of neural network such as might be used to implement the auto-associative memory unit 104 in FIG. 1. This example neural network is a recurrent neural network. However, this example network and the auto-associative memory unit 104 have a special architecture that makes them easier to train than many other types of recurrent neural networks.
In this discussion, a neural network comprises a network of nodes organized into layers, a layer of input nodes, zero or more inner or “hidden” layers of nodes, and a layer of output nodes. There is an input node associated with each input variable and an output node associated with each output variable. An inner layer may also be called a “hidden layer.” A given node in the output layer or in an inner layer is connected to one or more nodes in lower layers by means of a directed arc from the node in the lower layer to the given higher layer node. In this example network, there is also a directed arc from each output node back to the corresponding input node. A directed arc is an arc where direction matters, as opposed to undirected arcs. Note that there are only directed arc in the recurrent neural network shown in FIG. 4, as indicated by each arc between nodes having an arrow going from the lower level node to the higher level node, and from the output nodes to the input nodes. There are no undirected arcs in the recurrent neural network shown in FIG. 4.
The directed arcs are each associated with a trainable parameter, called its weight, which represents the strength of the connection from the lower node to the given higher node (or from an output node to its corresponding input node for the directed arcs from the output nodes to the input nodes). A trainable parameter is also called a “learned” parameter. Each node is also associated with an additional learned parameter called its “bias.” In a preferred embodiment, the weight associated with an arc from an output node to the corresponding input node implicitly has the value 1.0 and there is no learned parameter for these particular output-to-input arcs, as opposed to other arcs in the network that will have trainable (or learned) weights. Other parameters that control the learning process are called “hyperparameters.” The neural network illustrated in FIG. 4 has an input layer (“layer 0”), an output layer (“layer N” where N=6 in this example), and five hidden layers (layers 1 through N−1). In addition, the neural network in FIG. 4 and the auto-associative memory unit 104 has a set of target values, with a target objective for each output node.
A neural network in which there is no cycle of directed arcs leading from a node back to itself is called a “feed-forward” network. A neural network in which there is a cycle of directed arcs is called a “recurrent neural network.” In embodiments of the present invention, the cycle for the recurrent neural network are the directed arcs from the output nodes to their corresponding input nodes, as shown in FIG. 4. That is, in preferred embodiments of the present invention, and as explained herein, there are no directed arcs from a higher numbered layer to a lower numbered layer, except for the directed arcs from the output nodes to their corresponding input nodes.
For training purposes, a recurrent neural network R may be “unrolled” by making a sequence of copies of the neural network R(t) for each value oft in {0, 1, . . . , T}. In the case of the auto-associative memory unit 104 in FIG. 1 and the example neural network shown in FIG. 4, the index t counts the number of rounds of recursion. In this case, the value of T is the number of rounds of recursion until the recursion is stopped at convergence, at an infinite cycle, or by some other stopping criterion. Unrolling of a recurrent neural network R is depicted in FIG. 5. On the left side of the equation, a network R_toutputs value h_tin response to some input x_t. The recurrent nature of the network R_tis shown by the loop from the top of the network R_tto the bottom on the network R_t, depicting, as per the preferred embodiment of the present invention, the directed arcs from the output layer of the recurrent neural network to the associated nodes in the input layer.
The recurrent neural network R_ton the left side of the equation in FIG. 5 is unrolled into a sequence of copies of R_{0, . . . t}, as shown on the right side of the equation of FIG. 5. As per the preferred embodiment of the present invention, the recurrent nature of the recurrent neural network is depicted by the directed arc from the output layer of prior copy of R to the input layer of the next copy of R (e.g., the directed arc from the output layer of R₀to the input layer of R₁, and so on). Note that the directed arcs from the output layer of prior copy of R to the input layer of the next copy of R show the feed forward activation direction; back propagation for training is in the opposite direction.
The copies of R_{0, . . . t}, therefore collectively form a single large feed-forward neural network since each directed arc goes from a node in a higher numbered layer to a destination node in a lower numbered layer in copy R(t) not go to the node in its own copy R(t) (or from a node to itself or to a lower numbered destination node in its own layer), but rather have the directed arc go to the copy of the corresponding destination node in the next copy of the network R(t+1). Thus, in the unrolled network there are no cycles of directed arcs, so the unrolled network is a feed-forward network, as shown by the right side of FIG. 5.
In the auto-associative memory unit 104 or the network shown in FIG. 4, the only arcs that go from a higher numbered layer to a lower numbered layer are the directed arcs from the output nodes to the input nodes. In unrolling the recurrent neural network shown in FIG. 4 or the recurrent auto-associative memory unit 104, each copy R(t) also has its own copy of the target objective as well as the back propagation from the input to network R(t+1) back to the output from R(t). This feature is a special feature of this recurrent neural network architecture and is not present in unrolled recurrent neural networks in general.
A feed-forward neural network or an unrolled recurrent neural network may be trained using an iterative training process called stochastic gradient descent with a gradient estimate and learned parameter update for each minibatch of training data. An epoch of this iterative training process comprises a minibatch update for all the minibatches in the full batch of training data.
The estimate of the gradient of the objective function for each minibatch may be computed by accumulating an estimate of the gradient of the objective function for each training data item in the minibatch. The estimate of the gradient for a training data item may be computed by a feed-forward computation of the activation of each node in the network followed by a backwards computation of the partial derivatives of the objective function based on the chain rule of calculus. The backwards computation of the partial derivatives of the objective function is called “back propagation.”
Stochastic gradient descent, including the feed-forward computation, the back propagation of partial derivatives of an objective function, and unrolling a recurrent neural network are all well-known to those skilled in the art of training neural networks.
In the auto-associative memory unit 104 and in the neural network in FIG. 4, in the feed-forward network created by unrolling the recurrent neural network (see FIG. 5), each copy R(t) of the network receives a back propagated partial derivative from its own copy of the objective function as well as a back propagated partial derivative from the input nodes of the next copy of the network R(t+1). The partial derivatives from the input nodes of copy R(t+1) comprises the combined partial derivatives from objective t+1 and all higher numbered copies R(t+k) up to copy R(T).
In many recurrent neural network architectures, the unrolled feed-forward network is only an approximate model to the recursive neural network because in the activation computation of the nodes in a cycle in the recursive network can, in principle go around the cycle an infinite number of times. Furthermore, for many recurrent neural network architectures, during training a problem called “vanishing gradient” may occur for an unrolled recurrent neural network with too large a value of T. The magnitudes of the back propagated partial derivative may decrease by roughly a multiplicative factor for a factor less than 1.0 for each round of recursion, producing an exponential decay in the magnitudes of the partial derivatives.
Having an objective for each unrolled copy R(t) of the network R and accumulating the combined back propagated partial derivatives from higher numbered copies of R prevents this form “vanishing gradient.” In addition, the number of rounds of recursion in the auto-associative memory is limited to a finite number because of convergence or some other stopping criterion for the recursion.
In one general aspect, therefore, the present invention is directed to computer systems and computer-implemented methods for recursively training a content-addressable auto-associative memory. The system comprises a set of processor cores and computer memory that is in communication with the set of processor cores. The computer memory stores software that when executed by the set of processor cores, causes the set of processor cores to recursively train the content-addressable auto-associative memory system with a plurality of learned parameters and with a plurality of input examples, where each input example is represented by a plurality of input variables, such that: (i) the content addressable auto-associative memory system is trained to produce an output pattern for each of the input examples; and (ii) a quantity of the learned parameters for the content-addressable auto-associative memory is equal to the number of input variables times a quantity that is independent of the number of input variables. In various implementations, the software causes the set of processor cores to train the content-addressable auto-associative memory system such that the quantity of learned parameters for the content-addressable auto-associative memory system can be varied based on the number of input examples to be learned.
In another general aspect, the present invention is directed to computer systems and computer-implemented methods for recursively training a recurrent neural network with a plurality of input examples. In such embodiments, the computer system comprises a set of processor cores and computer memory in communication with the set of processor cores. The computer memory stores software that when executed by the set of processor cores, causes the set of processor cores to recursively train a recurrent neural network with a plurality of input examples, such that: (i) the recurrent neural network comprises a deep neural network that comprises N+1 layers, numbered 0, . . . , N, wherein N>3, and wherein layer 0 is an input layer and layer N is an output layer of the recurrent neural network, and wherein layers 1 to N−1 are between the input layer and the output layer; (ii) the recurrent neural network is trained to produce an output pattern for each of the input examples; (iii) a target for the output pattern for each input example is the input example; and (iv) the recurrent neural network comprises a plurality of directed arcs, wherein at least some of the directed arcs are between a node in one layer of the recurrent neural network and a node in another layer of the recurrent neural network.
In another implementation, the software stored by the computer memory causes the set of processor cores to recursively train the recurrent neural network such that: (i) the recurrent neural network is trained to produce an output pattern for each of the input examples; and (ii) a quantity of learned parameters for the recurrent neural network is equal to the number of input variables times a quantity that is independent of the number of input variables.
In various implementations, the software can cause the set of processor cores to train the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, such that the quantity of learned parameters for the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, can be varied based on the number of input examples to be learned. Also, the software can cause the set of processor cores to train the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, by back propagating partial derivatives of a loss function through the content-addressable auto-associative memory system or the recurrent neural network. Also, the software can cause the set of processor cores to train the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, by for each input example: randomly transforming the input example; and recursively providing the randomly transformed input example to the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, for training, until an output of the content-addressable auto-associative memory system converges to the input example. The random transformations of the input examples can comprise one or more of: translating the input example; rotating the input example; linearly transforming the input example; degrading the input example; and subsampling the input example.
In still further implementations, the computer memory may store software that when executed by the set of processor cores further causes the set of processor cores to train the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, with negative input examples. The negative input examples may comprise input examples where the output of the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, in operation, does not converge to an input example.
In still further implementations, at least some of the input examples are labeled examples that have, for each such input example, a classification category label such that the content-addressable auto-associative memory system or the recurrent neural network, as the case may be, is trained to act as a classifier. The classification category labels may comprise error-correcting encoding.
Based on the above description, it is clear that embodiments of the auto-associative memory system described herein can be content-addressable in the sense that an item in the memory may be retrieved by a description or example rather than by knowing the address of the item in memory. Further, the auto-associative memory is associative in that an item can be retrieved with a query based on an example of an associated item rather than by an example of the item itself. The auto-associative memory is also robust in that an item can be retrieved from a query that is a transformed, distorted, noisy version of the item to be retrieved. The auto-associative memory can be use, for example, to retrieve images, documents, acoustic files, etc. from inputs, which can be small pieces of the images, documents, acoustic files, etc.
The examples presented herein are intended to illustrate potential and specific implementations of the present invention. It can be appreciated that the examples are intended primarily for purposes of illustration of the invention for those skilled in the art. No particular aspect or aspects of the examples are necessarily intended to limit the scope of the present invention. Further, it is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for purposes of clarity, other elements. While various embodiments have been described herein, it should be apparent that various modifications, alterations, and adaptations to those embodiments may occur to persons skilled in the art with attainment of at least some of the advantages. The disclosed embodiments are therefore intended to include all such modifications, alterations, and adaptations without departing from the scope of the embodiments as set forth herein.

Claims

1. A computer system comprising:

a set of processor cores; and

computer memory in communication with the set of processor cores, wherein the computer memory stores software that when executed by the set of processor cores, causes the set of processor cores to recursively train a content-addressable auto-associative memory system with a plurality of learned parameters and with a plurality of input examples, wherein each input example is represented by a plurality of input variables, such that:

the content addressable auto-associative memory system is trained to produce an output pattern for each of the input examples; and

a quantity of the learned parameters for the content-addressable auto-associative memory is equal to the number of input variables times a quantity that is independent of the number of input variables.

2. The computer system of claim 1, wherein the software causes the set of processor cores to train the content-addressable auto-associative memory system such that the quantity of learned parameters for the content-addressable auto-associative memory system can be varied based on the number of input examples to be learned.

3. The computer system of claim 1, wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the content-addressable auto-associative memory system by back propagating partial derivatives of a loss function through the content-addressable auto-associative memory system.

4. The computer system of claim 1, wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the content-addressable auto-associative memory system by, for each input example:

randomly transforming the input example; and

recursively providing the randomly transformed input example to the content-addressable auto-associative memory system for training, until an output of the content-addressable auto-associative memory system converges to the input example.

5. The computer system of claim 4, wherein the computer memory stores software that when executed by the set of processor cores further causes the set of processor cores to train the content-addressable auto-associative memory system with negative input examples.

6. The computer system of claim 5, wherein the negative input examples comprise input examples where the output of the content-addressable auto-associative memory system, in operation, does not converge to an input example.

7-11. (canceled)

12. A computer system comprising:

a set of processor cores; and

computer memory in communication with the set of processor cores, wherein the computer memory stores software that when executed by the set of processor cores, causes the set of processor cores to recursively train a recurrent neural network with a plurality of input examples, such that:

the recurrent neural network comprises a deep neural network that comprises N+1 layers, numbered 0, . . . , N, wherein N>3, and wherein layer 0 is an input layer and layer N is an output layer of the recurrent neural network, and wherein layers 1 to N−1 are between the input layer and the output layer;

the recurrent neural network is trained to produce an output pattern for each of the input examples;

a target for the output pattern for each input example is the input example; and

the recurrent neural network comprises a plurality of directed arcs, wherein at least some of the directed arcs are between a node in one layer of the recurrent neural network and a node in another layer of the recurrent neural network.

13-15. (canceled)

16. The computer system of claim 12, wherein the software causes the set of processor cores to train the recurrent neural network such that the only directed arcs in the recurrent neural network from a higher numbered layer to a lower numbered layer are from a node in the output layer N to a node in the input layer 0.

17. The computer system of claim 16, wherein:

the output layer of the recurrent neural network comprise a plurality of output layer nodes;

the input layer of the recurrent neural network comprise a plurality of input layer nodes;

the quantity of input layer nodes equals the quantity of output layer nodes, such that each output layer node has one and only one corresponding input layer node; and

the only directed arcs in the recurrent neural network that are from a higher numbered layer to a lower numbered layer are directed arcs from the output layer to the input layer, wherein there is a directed arc from each output layer node to its associated input layer node.

18. The computer system of claim 12, wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by back propagating partial derivatives of a loss function through the recurrent neural network.

19. The computer system of claim 13, wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by back propagating partial derivatives of a loss function through the recurrent neural network.

20. The computer system of claim 12, wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by back propagating partial derivatives of a loss function through the recurrent neural network.

21. The computer system of claim 17, wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by, for each input example:

randomly transforming the input example; and

22. The computer system of claim 21, wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to transform an input example by performing a distortion on the input example that comprises a distortion selected from the group consisting of:

translating the input example;

rotating the input example;

linearly transforming the input example;

degrading the input example; and

subsampling the input example.

23. The computer system of claim 21, wherein the transformations of the input examples are controlled by one or more hyperparameters.

24. The computer system of claim 23, further comprising:

a second set of processor cores; and

second computer memory in communication with the second set of processor cores, wherein the second computer memory stores software that when executed by the second set of processor cores causes the second set of processor cores to implement a machine-learning learning coach that learns, through machine learning, values for the one or more hyperparameters.

25. The computer system of claim 21, wherein the computer memory stores software that when executed by the set of processor cores further causes the set of processor cores to train the recurrent neural network with negative input examples.

26. The computer system of claim 25, wherein the negative input examples comprise input examples where the output of the recurrent neural network, in operation, does not converge to an input example.

27. The computer system of claim 12, wherein at least some of the input examples are labeled examples that have, for each such input example, a classification category label such that the recurrent neural network is trained to act as a classifier.

28. The computer system of claim 27, wherein the classification category labels comprise error-correcting encoding.

29. The computer system of claim 12, wherein:

the input examples are digital images; and

the recurrent neural network is trained to, in operation, retrieve one of the digital images in response to receiving as input a portion of the digital image.

30. The computer system of claim 12, wherein:

the input examples are audio files; and

the recurrent neural network is trained to, in operation, retrieve one of the audio files in response to receiving as input a portion of the audio file.

31. The computer system of claim 12, wherein:

the input examples are document files; and

the recurrent neural network is trained to, in operation, retrieve one of the document files in response to receiving as input a portion of the document file.

32. A method comprising:

recursively train, by a computer system that comprises a set of processor cores, a content-addressable auto-associative memory system with a plurality of learned parameters and with a plurality of input examples, wherein each input example is represented by a plurality of input variables, such that:

33. The method of claim 32, wherein training the content-addressable auto-associative memory system comprises raining the content-addressable auto-associative memory system such that the quantity of learned parameters for the content-addressable auto-associative memory system can be varied based on the number of input examples to be learned.

34. The method of claim 32, wherein training the content-addressable auto-associative memory system comprises back propagating partial derivatives of a loss function through the content-addressable auto-associative memory system.

35. The method of claim 32, wherein training the content-addressable auto-associative memory system comprises, for each input example:

randomly transforming the input example; and

36. The method of claim 35, wherein training the content-addressable auto-associative memory system comprises training the content-addressable auto-associative memory system with negative input examples.

37. The method of claim 36, wherein the negative input examples comprise input examples where the output of the content-addressable auto-associative memory system, in operation, does not converge to an input example.

38. The method of claim 35, wherein at least some of the input examples are labeled examples that have, for each such input example, a classification category label such that the r content-addressable auto-associative memory system is trained to act as a classifier.

39. The method of claim 38, wherein the classification category labels comprise error-correcting encoding.

40. The method of claim 32, wherein:

the input examples are digital images; and

the content-addressable auto-associative memory system is trained to, in operation, retrieve one of the digital images in response to receiving as input a portion of the digital image.

41. The method of claim 32, wherein:

the input examples are audio files; and

the content-addressable auto-associative memory system is trained to, in operation, retrieve one of the audio files in response to receiving as input a portion of the audio file.

42. The method of claim 32, wherein:

the input examples are document files; and

the content-addressable auto-associative memory system is trained to, in operation, retrieve one of the document files in response to receiving as input a portion of the document file.

43. A method comprising:

training, recursively, by a computer system that comprises a set of processor cores, a recurrent neural network with a plurality of input examples, such that:

44. A method comprising:

the recurrent neural network is trained to produce an output pattern for each of the input examples; and

a quantity of learned parameters for the recurrent neural network is equal to the number of input variables times a quantity that is independent of the number of input variables.

45-46. (canceled)

47. The method of claim 43, wherein training the recurrent neural network comprises training the recurrent neural network such that the only directed arcs in the recurrent neural network from a higher numbered layer to a lower numbered layer are from a node in the output layer N to a node in the input layer 0.

48. The method of claim 47, wherein:

49. The method of claim 43, wherein training the recurrent neural network comprises back propagating partial derivatives of a loss function through the recurrent neural network.

50. The method of claim 44, wherein training the recurrent neural network comprises back propagating partial derivatives of a loss function through the recurrent neural network.

51. (canceled)

52. The method of claim 48, wherein training the recurrent neural network comprises, for each input example:

randomly transforming the input example; and

53. The method of claim 52, wherein transforming an input example by performing a distortion on the input example that comprises a distortion selected from the group consisting of:

translating the input example;

rotating the input example;

linearly transforming the input example;

degrading the input example; and

subsampling the input example.

54-55. (canceled)

56. The method of claim 52, wherein training the recurrent neural network comprises training the recurrent neural network with negative input examples.

57-62. (canceled)