JP2019519045A - Neural network and method of neural network training - Google Patents

Abstract

A neural network includes an input for receiving an input signal, and a synapse connected to the input and having correction weights organized in an array. Training images are either received as an array by input or are organized as such during network training. The network also has an output connected to at least one input via one synapse, and generates a neuron sum sequence by summing the correction weights selected from each synapse connected to the respective neuron , Including neurons. In addition, the network includes a controller that receives the desired images in the form of an array, determines the deviation of the neuron sum array from the desired output value array, and generates a deviating array. The controller uses the deviance array to change the modified weight array. Summing the modified correction weights and determining the neuron sum arrangement reduces the problem deviation and produces a trained correction weight array for simultaneous network training.

Description

This application claims the benefit of US Ser. No. 15 / 449,614, filed Mar. 3, 2017, and US Ser. No. 15 / 178,137, filed Jun. 9, 2016. Claim the benefits of Each of these applications is a continuation-in-part of US Patent Application No. 14 / 862,337, filed on September 23, 2015, which application is filed on March 6, 2015. No. PCT / US2015 / 19236, which is a continuation of US Provisional Patent Application No. 61 / 949,210, filed March 6, 2014, and the United States filed January 22, 2015. Provisional Patent Application No. 62 / 106,389, further claiming the benefit of US Provisional Patent Application No. 62 / 173,163, filed on Jun. 9, 2015, the entire contents of that application Is likewise incorporated herein by reference.

  The present disclosure relates to artificial neural networks and methods of training them.

  In machine learning, the term "neural network" generally refers to the overall design or structure of a computer system or microprocessor, including software and / or computer architecture, ie, the hardware and software required to execute it. Point to. Artificial neural networks may be a group of statistical learning algorithms inspired by biological neural networks, also known as the central nervous system of animals, in particular the brain. Artificial neural networks are mainly used to estimate or approximate generally unknown functions that may depend on a large number of inputs. Such neural networks have been used for a wide variety of tasks that are difficult to solve using ordinary rule-based programming, including computer vision and speech recognition.

  Artificial neural networks are generally presented as a system of "neurons" that can calculate values from inputs, and, due to their adaptability, have the capabilities of machine learning and pattern recognition. Each neuron often connects to several inputs through synapses with synaptic weights.

  Neural networks are not programmed as typical software and hardware but are trained. Such training is typically through analysis of a sufficient number of representative cases to statistically or algorithmically select synaptic weights such that a given set of input images correspond to a given set of output images. Achieved by A frequent criticism of classical neural networks is that a great deal of time and other resources are often needed for their training.

  Various artificial neural networks are described in U.S. Pat. Nos. 4,979,124, 5,479,575, 5,493,688, 5,566,273, 5,682,503. Nos. 5,870,729, 7,577,631 and 7,814,038.

  In one embodiment, the neural network includes a plurality of inputs to a neural network configured to receive a training image. Either the training image is received as a training input value array by multiple inputs, or is organized during training of the neural network, ie, after being received by multiple inputs, as a training input value array It is. Neural networks also include multiple synapses. Each synapse is connected to one of a plurality of inputs and includes a plurality of correction weights. Each correction weight is defined by a weight value, and the correction weights of the plurality of synapses are organized into a correction weight array.

  The neural network further includes a plurality of neurons. Each neuron has at least one output and is connected to at least one of the plurality of inputs via at least one of the plurality of synapses. Each neuron is configured to add the weight value of the correction weight corresponding to each synapse connected to the respective neuron, whereby the plurality of neurons generate a neuron sum sequence. The neural network also includes a controller configured to receive the desired image organized as the desired output value array.

  The controller is also configured to determine a departure of the neuron sum sequence from the desired output value sequence and to generate a departure sequence. The controller is further configured to change the modified weight sequence using the determined deviation sequence. Summing the modified corrected weight values to determine the neuron sum sequence reduces the deviation of the neuron sum sequence from the desired output value array. That is, it compensates for errors generated by the neuron network during training and produces a trained correction weight array, thereby facilitating simultaneous or parallel training of the neural network.

  In a trained neural network, multiple inputs to the neural network may be configured to receive an input image. Such an input image may either be received as an input value array or may be organized as an input value array during recognition of the image by a neural network. Each synapse may include a plurality of trained correction weights of the trained correction weight sequence. In addition, each neuron may be configured to add up the weight values of the trained correction weights corresponding to each synapse connected to the respective neuron, whereby a plurality of neurons are recognized An image array is generated, which results in the recognition of such an input image.

  The neural network may also include a set of distributors. In such embodiments, the set of distributors may be configured to organize each of the training image and the input image as a respective training input value array and input value array. Such a set of distributors may be operatively connected to a plurality of inputs for receiving respective training images and input images.

  The controller may further be programmed with an array of target deviations of the neuron sum array from the desired output value array. Further, the controller may be configured to complete training of the neural network when the deviation of the neuron sum sequence from the desired output value sequence falls within the tolerance of the target deviation sequence.

  The training input value array, the input value array, the correction weight array, the neuron sum array, the desired output value array, the deviation array, the trained correction weight array, the recognized image array, and the target deviation array are each a training input value It may be organized as a matrix, an input value matrix, a correction weight matrix, a neuron sum matrix, a desired output value matrix, a deviation matrix, a trained correction weight matrix, a recognized image matrix, and a target deviation matrix.

  The neural network may further include a plurality of data processors. In such embodiments, the controller further partitions at least one of the respective input value matrix, the training input value matrix, the modified weight matrix, the neuron sum matrix, and the desired output value matrix into respective submatrices And may be configured to communicate the plurality of derived submatrices to a plurality of data processors for separate parallel mathematical operations with them. This partitioning of any of the problem matrices into respective submatrices increases the speed of either simultaneous or parallel data processing, as well as either image recognition of the input value matrix or training of the neural network. make it easier. Such simultaneous or parallel data processing also enables neural network scalability.

  The controller may modify the modified weighting matrix by applying algebraic matrix operations to the training input value matrix and the modified weighting matrix to train the neural network.

  The mathematical matrix operation may include determining the mathematical product of the training input value matrix and the modified weight matrix, thereby forming a weight matrix of the current training epoch.

  The controller may also be configured to subtract the neuron sum matrix from the desired output value matrix to generate a matrix of neuron sum deviations. In addition, the controller may be configured to divide the matrix of neuron sum deviations by the number of inputs connected to each neuron to generate a matrix of deviations for each neuron input.

  The controller may also be configured to determine the number of times each correction weight has been used during one training epoch of the neural network. The controller may be further configured to form an average deviation matrix for one training epoch using the determined number of times each correction weight was used during one training epoch. Further, the controller may be configured to add the mean deviation matrix for one training epoch to the modified weight matrix, thereby generating a trained modified weight matrix, and completing one training epoch.

  Also disclosed is a method of operating such a neural network, ie a method for training and image recognition.

  Additionally disclosed are non-transitory computer readable storage devices for operating artificial neural networks and devices for operating artificial neural networks.

  In another embodiment, the neural network includes a plurality of network inputs, whereby each input is configured to receive an input signal having an input value. The neural network also includes a plurality of synapses, each synapse being connected to one of a plurality of inputs, including a plurality of correction weights, each correction weight being by a memory element holding a respective weight value. Is established. The neural network further includes a set of distributors. Each distributor is operatively connected to one of the plurality of inputs for receiving the respective input signal, and selects one or more correction weights from the plurality of correction weights in correlation with the input value. It is configured to The neural network also includes a set of neurons. Each neuron has at least one output and is connected to at least one of the plurality of inputs via one of the plurality of synapse groups, selected from each synapse connected to the respective neuron The weighting values of the correction weights are summed up to generate a neuron sum. The output of each neuron provides the respective neuron sum to establish the working output signal of the neural network.

  The neural network also receives a desired output signal having a value, determines a deviation of the neuron sum from the desired output signal value, and uses the determined deviation to correct each of the corrections established by the corresponding memory element. It may include a weight correction calculator configured to change weight values. In such a case, summing the modified corrected weight values and determining the neuron sum minimizes the deviation of the neuron sum from the desired output signal value, thereby producing a trained neural network. It is intended to be.

  The trained neural network may be configured to receive supplemental training using only the supplemental input signal having values and the corresponding supplemental desired output signal.

  Either during training or after supplemental training of the neural network, each of the plurality of synapses can be configured to accept one or more additional correction weights established by the respective memory element.

  The neural network may be configured to remove one or more correction weights established by the respective memory element from the respective synapse during or after the training of the neural network. Such elimination of some correction weights may allow the neural network to hold only the number of memory elements needed to operate the neural network.

  The neural network is configured to receive at least one of an additional input, an additional synapse, and an additional neuron before or during training of the neural network, thereby extending the operating parameters of the neural network. obtain.

  The neural network may be configured to delete at least one of the input, synapse and neurons before, during or after training of the neural network. Such ability to eliminate neural network elements not used by the network is intended to simplify the structure and to change the operating parameters of the neural network without loss of network output quality.

  Each memory element may be established by an electrical device characterized by electrical and / or magnetic properties configured to define a respective weight value. Such characteristics may be resistance, impedance, capacitance, magnetic field, induction, electric field strength, etc. The respective electrical and / or magnetic properties of each device may be configured to be changed during training of the neural network. Additionally, the weight correction calculator may change the respective correction weight value by changing at least one of each of the electrical and magnetic properties of the corresponding electrical device.

  The electric device may be a resistor, a resonator, a memristor, a transistor, a capacitor, a field effect transistor, a photo-resistor such as a light-dependent resistor (LDR), or a magnetic dependent resistor (MDR). Can be configured as one of

  Each memory element is established by the block of electrical resistors and may include a selection device configured to select one or more electrical resistors from the block using the determined deviation and to establish each modified weight.

  The block of electrical resistors may further include electrical capacitors. In other words, each memory element can be established by a block having both an electrical resistor and an electrical capacitor. At this time, the selection device may be further configured to select capacitors using the determined deviation and to establish each correction weight.

  The neural network may be configured as one of an analog network, a digital network, and a digital-analog network. In such a network, at least one of the plurality of inputs, the plurality of synapses, the memory element, the set of distributors, the set of neurons, the weight correction calculator, and the desired output signal may be in analog format, digital format, And may be configured to operate in a digital-analog format.

  If the neural network is configured as an analog network, each neuron may be established by a wire, or one of serial and parallel communication channels, such as a serial bus or a parallel bus.

  The weight correction calculator may be established as a set of differential amplifiers. Furthermore, each differential amplifier may be configured to generate a respective correction signal.

  Each of the distributors may be a demultiplexer configured to select one or more of the correction weights from the plurality of correction weights in response to the received input signal.

  Each distributor may be configured to convert the received input signal into a binary code and to correlate to the binary code to select one or more correction weights from the plurality of correction weights.

  The neural network may be programmed in an electronic device having a memory, each memory element being stored in a memory of the electronic device.

  Also disclosed is a method of operating a utility neural network. The method includes processing the data via the utility neural network with the modified correction weight values established during the training by a separate similar neural network. The method also includes establishing the operational output signal of the utility neural network using the modified correction weight values established by the separate similarity neural network.

  For the use of the modified correction weight values by the utility neural network, the separate similar neural network receives the training input signal having the training input value via the input to the neural network, and the training input signal Communicating with a splitter operatively connected to the input, and, via the splitter, correlating to the training input value to select one or more of the correction weights from the plurality of correction weights, Each correction weight is defined by a weight value and is located on a synapse connected to the input, selecting and connected to the input via the synapse, selected correction via a neuron having at least one output Summing the weights and generating the neuron sum, and the value via the weight correction calculator Receiving the desired output signal having, determining the deviation of the neuron sum from the desired output signal value via the weight correction calculator, and using the determined deviation via the weight correction calculator Changing the respective modified weight values and establishing the modified corrected weight values, summing the modified corrected weight values and determining the neuron sum from the desired output signal value Training was done through establishing, which minimizes the deviation of the sum, thereby training the neural network.

  The utility neural network and the trained separate similar neural network may include matched neural network structures including multiple inputs, correction weights, distributors, neurons, and synapses.

  In each of the utility neural network and the trained separate similar neural network, each correction weight may be established by a memory element holding a respective weight value.

  The above features and advantages, as well as other features and advantages of the present disclosure, will be better understood by the following detailed description of embodiments and best modes for practicing the disclosure in conjunction with the accompanying drawings and the appended claims. It will be readily apparent from

FIG. 1 is a schematic diagram of a prior art classical artificial neural network. FIG. 2 is a schematic diagram of a "progressive neural network" (p-net) having a plurality of synapses, a set of distributors, and a plurality of modified weights associated with each synapse. FIG. 3A is a schematic diagram of a portion of the p-net shown in FIG. 2 with multiple synapses and one synapse weight located upstream of each splitter. FIG. 3B is a schematic diagram of a portion of the p-net shown in FIG. 2 with a plurality of synapses and a set of synapse weights positioned downstream of each of the plurality of modification weights. FIG. 3C shows the p shown in FIG. 2 with a plurality of synapses, a single synapse weight located upstream of each splitter, and a set of synapse weights located downstream of each of the plurality of modification weights. It is the schematic of a part of net | network. FIG. 4A is a schematic of the portion of the p-net shown in FIG. 2 having a single splitter for all synapses of a given input and one synapse weight located upstream of each splitter. FIG. FIG. 4B shows the p-net shown in FIG. 2 with a single splitter for all synapses of a given input and a set of synapse weights located downstream of each of a plurality of modification weights. It is the schematic of a part. FIG. 4C has a single splitter for all synapses of a given input and is located downstream of one synapse weight located upstream of each splitter and each of a plurality of correction weights 3 is a schematic view of a portion of the p-net shown in FIG. 2 with the set of synapse weights. FIG. 5 is a schematic diagram of the division of the range of input signal values into the individual intervals in the p-net shown in FIG. FIG. 6A is a schematic diagram of an embodiment of a distribution of values of influence coefficients of the modified weights in the p-net shown in FIG. FIG. 6B is a schematic diagram of another embodiment of the distribution for the values of the influence coefficients of the modified weights in the p-net shown in FIG. FIG. 6C is a schematic diagram of yet another embodiment of the distribution of influence coefficient values of the modified weights in the p-net shown in FIG. FIG. 7 is a schematic representation of the input image for the p-net shown in FIG. 2 and one corresponding table representing the image in the form of a digital code and another corresponding table representing the same image as a set of respective intervals FIG. FIG. 8 is a schematic diagram of an embodiment of the p-net shown in FIG. 2 trained for the recognition of two different images, wherein the p-net recognizes pictures that include some features of each image Fig. 1 is a schematic view, configured as follows. FIG. 9 is a schematic diagram of an embodiment of the p-net shown in FIG. 2 with an example of the distribution of synapse weights around "central" neurons. FIG. 10 is a schematic diagram of an embodiment of the p-net shown in FIG. 2 showing a uniform distribution of training deviations among the correction weights. FIG. 11 is a schematic diagram of an embodiment of the p-net shown in FIG. 2 that utilizes changes in correction weights during p-net training. FIG. 12 shows that the basic algorithm generates a first-order set of output neuron sums, and the generated set generates several "winner" sums having either a retained value or an increased value. FIG. 5 is a schematic diagram of an embodiment of the p-net shown in FIG. 2 used and the remaining sum contribution is nullified. FIG. 13 is a schematic view of an embodiment in which the p-net shown in FIG. 2 recognizes a complex image having a plurality of image elements. FIG. 14 is a schematic diagram of a model for object oriented programming for the p-net shown in FIG. 2 using Unified Modeling Language (UML). FIG. 15 is a schematic diagram of the overall formation sequence of the p-net shown in FIG. FIG. 16 is a schematic diagram of representative analysis and preparation of data for the formation of the p-net shown in FIG. FIG. 17 is a schematic diagram of representative input generation enabling interaction of the p-net shown in FIG. 2 with input data during training and p-net application. FIG. 18 is a schematic diagram of an exemplary creation of a neuron unit for the p-net shown in FIG. FIG. 19 is a schematic diagram of a representative preparation of each synapse connected with a neuron unit. FIG. 20 is a schematic diagram of the training of the p-net shown in FIG. FIG. 21 is a schematic view of training of neuron units in the p-net shown in FIG. FIG. 22 is a schematic diagram of the extension of the neuron sum during training of the p-net shown in FIG. FIG. 23 is a flow diagram of a method used to train the p-nets shown in FIGS. FIG. 24 is a schematic diagram of a particular embodiment of a p-net having each of a plurality of correction weights established by a memory element, wherein the p-net is shown in the process of network training. FIG. 25 is a schematic diagram of a particular embodiment of a p-net having each of a plurality of correction weights established by a memory element, wherein the p-net is shown in the process of image recognition. FIG. 26 is a schematic diagram of a representative p-net using memristor during the first phase of training. FIG. 27 is a schematic diagram of a representative p-net using memristor during the second phase of training. FIG. 28 is a schematic diagram of dual parallel branch of memristor in a typical p-net. FIG. 29 is a schematic diagram of a representative p-net using a resistor. FIG. 30 is a schematic diagram of one embodiment of a memory element configured as a resistor in a p-net. FIG. 31 is a schematic diagram of another embodiment of a memory element configured as a resistor in a p-net. FIG. 32 is a schematic diagram of another embodiment of a memory element configured as a variable impedance in a p-net. FIG. 33 is a flow diagram of a method used to operate the neural network shown in FIGS. 2-22 and 24-32. FIG. 34 is a diagram of a "progressive neural network" (p-net) with multiple synapses and multiple modified weights associated with each synapse, in accordance with the present disclosure; FIG. 35 is a diagram of a p-net during training according to the present disclosure. FIG. 36 is a diagram of a p-net during image recognition according to the present disclosure. FIG. 37 is a flow diagram of a method for operating the neural network shown in FIGS. 34-36.

  A classical artificial neural network 10, as shown in FIG. 1, typically comprises a neuron 18 and a neuron comprising an input device 12, a synapse 14 with synaptic weights 16, an adder 20 and an activation function device 22. An output 24 and a weight correction calculator 26 are included. Each neuron 18 is connected to two or more input devices 12 through a synapse 14. The value of synapse weight 16 is generally expressed using electrical resistance, conductivity, voltage, charge, magnetic properties, or other parameters.

Supervised training of the classical neural network 10 is generally based on the application of a set of training pairs 28. Each training pair 28 generally consists of an input image 28-1 and a desired output image 28-2, also known as a teacher signal. The training of the classical neural network 10 is typically performed as follows. An input image in the form of a set of input signals (I ₁ -I _m ) enters the input device 12 and is transferred to a synapse weight 16 having an initial weight (W ₁ ). The value of the input signal is modified by the weight, typically by multiplying or dividing the value of each signal (I ₁ -I _m ) by the respective weight. From the synapse weights 16 each modified input signal is forwarded to a respective neuron 18. Each neuron 18 receives a set of signals from the group of synapses 14 associated with the neuron 18 in question. The adder 20 contained within the neuron 18 is modified by the weights and sums all the input signals received by the neuron in question. The activation function device 22 receives each obtained neuron sum, modifies the sum according to the mathematical function, and thus forms each output image as a set of neuron output signals (ΣF ₁ ... FF _n ).

The resulting neuron output image, defined by the neuron output signals (ΣF ₁ ... FF _n ), is compared by the weight correction calculator 26 with a predetermined desired output image (O _{1 to} O _n ). Based on the determined difference between the obtained neuron output image FF _n and the desired output image O _n , a correction signal for changing the synapse weights 16 is formed using a preprogrammed algorithm. After all the synapse weights 16 have been corrected, the set of input signals (I ₁ -I _m ) is reintroduced into the neural network 10 and a new correction is made. Difference resulting neuron output image .SIGMA.F _n and the desired output image O _n until it is determined that some is less than the predetermined error, the above cycle is repeated. One cycle of network training with all individual images is commonly identified as a "training epoch". Generally, at each training epoch, the magnitude of the error is reduced. However, depending on the number of individual inputs (I _{1 to} I _m ), and the number of inputs and outputs, the training of the classical neural network 10 can possibly be hundreds of thousands in size, It may require a huge number of training epochs.

  Various classical neural networks exist, including Hopfield networks, restricted Boltzmann machines, radial basis function networks, and recurrent neural networks. The specific task of classification and clustering requires a specific type of neural network, a self-organizing map using only the input image as network input training information, whereas the desired output image corresponding to the specific input image Are formed directly during the training process based on a single winner neuron having an output signal with a maximum value.

  As mentioned above, one of the main concerns for existing classical neural networks, such as neural network 10, is that these trainings may require an enormous amount of time to be successful. Some additional concerns for classical networks may be the large consumption of computing resources. This will result in the need for a high performance computer. A further concern is the inability to increase the size of the network without complete retraining of the network, and the tendency to be prone to phenomena such as "network paralysis" and "freezes at the minima". This makes it difficult to predict whether a particular neural network can be trained with a given set of images in a given sequence. There may also be limitations associated with specific sequencing where images are introduced during training. In this case, changing the order of introduction of training images may cause the network to freeze and prevent further training of already trained networks.

Referring to the remaining figures, like reference numerals refer to like components and FIG. 2 shows a schematic of a progressive neural network, hereinafter "progressive network" or "p-net" 100. The p-net 100 includes a plurality of inputs 102, or sets of inputs 102, of the p-net. Each input 102 is configured to receive an input signal 104. Here, in FIG. 2, the input signals are represented as I ₁ , I ₂ ... I _m . Each input signal I ₁ , I ₂ ... I _m represents some characteristic of the input image 106, eg, magnitude, frequency, phase, signal declination, or a value associated with a different part of the input image 106. Each input signal 104 has an input value, and collectively, the plurality of input signals 104 generally describe the input image 106.

  Each input value may be in a range of values from -− to + ∞, and may be set in digital and / or analog form. The range of input values may depend on the set of training images. In the simplest case, the range of input values could be the difference between the smallest value and the largest value of the input signal for all training images. For practical reasons, the range of input values may be limited by eliminating input values that are considered to be excessively high. For example, such limitation of the range of input values can be achieved through known statistical methods for variance reduction, such as importance sampling. Another example of limiting the range of input values is the specification of all signals below a predetermined minimum level to a specific minimum, and to all signals above a predetermined maximum level at a specific maximum. May be specified.

  The p-net 100 also includes a plurality of synapses 118 or a set of synapses 118. Each synapse 118 is connected to one of a plurality of inputs 102, as shown in FIG. 2, includes a plurality of correction weights 112, and may also include a synapse weight 108. Each correction weight 112 is defined by a respective weight value. The p-net 100 also includes a set of distributors 114. Each distributor 114 is operatively connected to one of a plurality of inputs 102 for receiving a respective input signal 104. In addition, each distributor 114 is configured to select one or more correction weights from the plurality of correction weights 112 in correlation to the input value.

  p-net 100 further includes a set of neurons 116. Each neuron 116 has at least one output 117 and is connected to at least one of the plurality of inputs 102 via one synapse 118. Each neuron 116 sums up or adds the modified weight values of the modified weights 112 selected from each synapse 118 connected to the respective neuron 116, thereby generating and outputting the neuron sum 120 separately described as n n. Is configured as. A separate splitter 114 may be used for each synapse 118 of a given input 102, as shown in FIGS. 3A, 3B and 3C, or as shown in FIGS. 4A, 4B and 4C. As such, a single splitter may be used for all such synapses. During formation or setup of the p-net 100, all the correction weights 112 are assigned initial values that may change during the process of p-net training. The initial values of the correction weights 112 can be assigned as in the case of the classical neural network 10, for example, the weights can be randomly selected, predetermined templates calculated with the aid of predetermined mathematical functions And so on.

  The p-net 100 also includes a weight correction calculator 122. Weight correction calculator 122 is configured to receive a desired or predetermined output signal 124 that has signal values and that represents a portion of output image 126. The weight correction calculator 122 also determines from the value of the desired output signal 124 a deviation 128 of the neuron sum 120, also known as a training error, and changes the respective correction weight value using the determined deviation 128. Is configured as. Then, summing the modified correction weights and determining the neuron sum 120 minimizes the problem of the neuron sum deviation from the desired output signal 124 value, and thus training the p-net 100. It is effective for

  For analogy with the classical network 10 described with respect to FIG. 1, the deviation 128 can also be described as a training error between the determined neuron sum 120 and the value of the desired output signal 124. As compared to the classical neural network 10 described with respect to FIG. 1, in the p-net 100, the input value of the input signal 104 changes only during general network setup and changes during p-net training I will not. Instead of changing the input value, training of the p-net 100 is provided by changing the value 112 of the correction weight 112. In addition, while each neuron 116 includes a summation function in which the neurons sum correction weight values, the neurons 116 include activation functions such as those provided by the activation function device 22 in the classical neural network 10. It is not required and is in fact characterized by the absence of an activation function.

  In the classical neural network 10, weight correction during training is achieved by changing the synaptic weights 16, while in the p-net 100, the correction weight values 112 are changed as shown in FIG. The result is a corresponding weight correction. Each correction weight 112 may be included in a weight correction block 110 located on all or part of the synapse 118. In neural network computer emulation, each synapse weight and correction weight may be represented by either a digital device such as a memory cell and / or an analog device. In neural network software emulation, the values of the correction weights 112 can be provided via a suitable programmed algorithm, while in hardware emulation the known method for memory control is used Will be able to

  In the p-net 100, the deviation 128 of the neuron sum 120 from the desired output signal 124 can be represented as a mathematically calculated difference between the two. In addition, the generation of each modified correction weight 112 may include an allocation of the calculated difference to each correction weight used to generate the neuron sum 120. In such embodiments, the generation of each modified correction weight 112 will allow the neuron sum 120 to converge to the desired output signal value within a small number of epochs, in some cases It only needs a single epoch and trains the p-net 100 fast. In a particular example, the distribution of the mathematical differences among the correction weights 112 used to generate the neuron sum 120 is determined by dividing the determined differences by the respective correction weights used to generate the respective neuron sum 120. It may include dividing equally between.

In a separate embodiment, determining the deviation 128 of the neuron sum 120 from the desired output signal value may include dividing the desired output signal value by the neuron sum, thereby generating a deviation factor. In such a particular example, changing each modified correction weight 112 includes multiplying each correction weight used to generate neuron sum 120 by a deviation factor. Each distributor 114 may be further configured to assign a plurality of influence factors 134 to a plurality of correction weights 112. In the present embodiment, each influence factor 134 may be assigned to one of the plurality of correction weights 112 at some predetermined rate to generate a respective neuron sum 120. In order to correspond to each respective correction weight 112, each influence coefficient 134 may be given the name " _{, i, d, n} " as shown in the figure.

Each of the plurality of influence factors 134 corresponding to a particular synapse 118 is defined by a respective influence distribution function 136. The influence distribution function 136 may either be the same for all influence coefficients 134 or only for a plurality of influence coefficients 134 corresponding to a particular synapse 118. Each of the plurality of input values may be received within the range 137 of values divided into intervals or subdivisions "d" according to the interval distribution function 140, whereby each input value is received within the respective interval "d" And each correction weight corresponds to one of such intervals. Each distributor 114 selects the respective interval "d" using the respective received input value, and the respective influence coefficients 134 are corrected weights 112 corresponding to the respective selected intervals "d". And W _{i, d + 1, n} or W _{i, d-1, n,} etc. may be assigned to at least one correction weight corresponding to the interval adjacent to the selected interval. In another non-limiting example, the predetermined percentage of influence coefficients 134 may be defined according to a statistical distribution.

  Generating the neuron sum 120 first assigns each influence coefficient 134 to each correction weight 112 according to the input value 102, and then, for the problem influence coefficient, the value of each used correction weight 112 It may include multiplying. Next, through each neuron 116, the individual products of the correction weights 112 and the assigned influence coefficients 134 are summed over all synapses 118 connected thereto.

  Weight correction calculator 122 may be configured to apply respective influence factors 134 to generate respective modified correction weights 112. Specifically, the weight correction calculator 122 generates a neuron sum 120 for a portion of the calculated mathematical difference between the neuron sum 120 and the desired output signal 124 according to the proportion established by the respective influence factor 134 Can be applied to each of the correction weights 112 used to In addition, the mathematical differences divided between the correction weights 112 used to generate the neuron sum 120 can be further divided by the respective influence factors 134. The result of the division of the neuron sum 120 by the respective influence factor 134 may then be added to the correction weights 112 to converge the neuron sum 120 to the desired output signal value.

  Typically, formation of the p-net 100 will be performed before training of the p-net starts. However, in separate embodiments, during training, if the p-net 100 receives an input signal 104 without an initial correction weight, an appropriate correction weight 112 may be generated. In such a case, a particular splitter 114 will determine the appropriate spacing "d" for a particular input signal 104, and the group of correction weights 112 having an initial value is a given input. 102 will be generated for a given interval "d" and all respective neurons 116. In addition, corresponding influence factors 134 may be assigned to each newly generated correction weight 112.

Each modification weight 112 may be defined by a set of indices configured to identify the location of each respective modification weight on p-net 100. The set of indices specifically identifies the input index "i" configured to identify the correction weights 112 corresponding to the particular input 102, the above selected intervals for each of the correction weights And the neuron index "n" configured to identify the correction weight 112 corresponding to a particular neuron 116 by the name "W _{i, d, n} ". Therefore, each correction weight 112 corresponding to a particular input 102 is assigned a particular index "i" within the subscript to indicate the location of the problem. Similarly, each correction weight "W" corresponding to a particular neuron 116 and a respective synapse 118 has a particular index "n" in the subscript to indicate the position of the problem of the correction weight on p-net 100. And "d" are assigned. The set of indices may also include an access index "a" configured to count the number of times each modification weight 112 has been accessed by the input signal 104 during p-net 100 training. In other words, access index "a" counts the input signal each time a particular interval "d" and the respective correction weight 112 are selected in correlation to the input value from the plurality of correction weights for training. To be incremented. The access index "a" may be used to further specify or define the current status of each modified weight by adopting the name "W _{i, d, n, a} ". Each of the indexes “i”, “d”, “n”, and “a” may be numerical values in the range of 0 to + ∞.

FIG. 5 shows various possibilities of dividing the range of the input signal 104 into the intervals d ₀ , d ₁ ... D _m . The particular spacing distribution may be uniform or linear. This may be achieved, for example, by specifying all intervals "d" by the same size. All input signals 104 having a respective input signal value below a predetermined minimum level can be considered as having a value of 0, while having a respective input signal value greater than a predetermined maximum level. All input signals can be assigned to such highest levels, as also shown in FIG. The particular spacing distribution may also be non-uniform or non-linear, such as symmetric, asymmetric or unlimited. The non-linear distribution of the interval "d" is considered that the range of the input signal 104 is considered to be impractically large, and certain parts of the range, such as at the beginning, middle, or end of the range, are considered most important. It may be useful when it may be included. Specific interval distributions can also be described by random functions. The above examples are all of non-limiting nature, as other variations of the spacing distribution are possible.

  The number of intervals “d” within the selected range of input signal 104 may be increased to optimize p-net 100. Such optimization of p-net 100 may be desirable, for example, with increasing training complexity of input image 106. For example, for multi-color images, more spacing may be required compared to monochrome images, and for complex adornments, more spacing may be required than simple graphics. For accurate recognition of images with complex color gradients, an increased number of intervals may be required as compared to the image described by the contour, as is the case with larger total training images. Also, if the magnitude of the noise is high, if the variance in the training image is high, and the consumption of computational resources is excessive, then a reduction in the number of intervals "d" may also be required.

  Depending on the task being processed by the p-net 100 or the type of information, eg visual data or character data, data from different types of sensors, different numbers of intervals and types of their distribution may be specified. For each input signal value interval "d", a corresponding correction weight of a given synapse with index "d" may be assigned. Thus, a particular interval "d" is an index "i" associated with a given input, an index "d" associated with a given interval, and all values for an index "n" from 0 to n. Will include all the correction weights 112. In the process of training p-net 100, splitter 114 defines each input signal value, thus associating the input signal in question 104 with the corresponding interval "d". For example, if there are ten equal intervals "d" in the range of 0 to 100 input signals, then an input signal having a value of 30 to 40 would be associated with interval 3, ie "d '= 3.

For all correction weights 112 of each synapse 118 connected with a given input 102, the distributor 114 may assign the value of the influence factor 134 according to the interval "d" associated with the particular input signal. Divider 114 may also assign values of influence coefficient 134 according to a predetermined distribution of values of influence coefficient 134 (shown in FIG. 6), such as sine waves, normal, logarithmic distribution curves, etc., or according to a random distribution function. In many cases, the sum or integral of the influence coefficients 134 or С _{i, d, n} for the particular input signal 102 associated with each synapse 118 will have a value of 1 (1).
Syn _Synapse С _{i, d, n} = 1 or ∫ _Synapse С _{i, d, n} = 1 [1]
In the simplest case, the correction weight 112, which most closely corresponds to the input signal value, may be assigned a value of 1 (1) for the influence factor 134 (С _{i, d, n} ), while the other intervals The correction weights for can receive a value of 0 (0).

The p-net 100 focuses on reducing the use of period and other resources during training of the p-net as compared to the classical neuron network 10. Although some of the elements disclosed herein as part of p-net 100 are designated by particular names or identifiers known to those familiar with classical neural networks, particular names Are used for brevity and may be used differently from their counterparts in classical neural networks. For example, the synaptic weights 16 controlling the magnitude of the input signals (I ₁ -I _m ) are defined during the process of the general setup of the classical neural network 10 and are changed during the training of the classical network Be done. On the other hand, training of the p-net 100 is achieved by changing the correction weights 112, while the synapse weights 108 do not change during training. In addition, as described above, each of the neurons 116 includes a summation or summation component but does not include the activation function device 22 that is typical of the classical neural network 10.

  In general, p-net 100 trains each neuron unit 119, including each neuron 116 and all connected synapses 118, including a particular neuron and all respective synapses 118, and the correction weights 112 connected with the neuron in question. Are trained by doing. Thus, training of the p-net 100 involves changing the correction weights 112 that contribute to each neuron 116. Changes to the correction weights 112 are made based on a group training algorithm included in the method 200 disclosed in detail below. In the algorithm of the present disclosure, a training error or deviation 128 is determined for each neuron, based on which a correction value is determined and a weight 112 used in determining the sum obtained by each respective neuron 116. Assigned to each of The introduction of such corrections during training is intended to reduce the deviation 128 for the neuron 116 in question to zero. During training with additional images, new errors associated with previously utilized images may reappear. In order to eliminate such additional errors, the errors for all training images of the entire p-net 100 may be calculated after completion of one training epoch, such errors being greater than a predetermined value. In some cases, one or more additional training epochs may be performed until the error is less than the target or predetermined value.

FIG. 23 illustrates a method 200 for training p-net 100 as described above with reference to FIGS. Method 200 begins at frame 202. At frame 202, the method includes receiving, via input 102, an input signal 104 having an input value. Following frame 202, the method proceeds to frame 204. At frame 204, the method includes communicating the input signal 104 to a splitter 114 operatively connected to the input 102. In either frame 202 or frame 204, method 200 may include defining each correction weight 112 by a set of indices. As described above with respect to the structure of p-net 100, the set of indices may include an input index "i" configured to identify the correction weight 112 corresponding to input 102. The set of indices may also be configured to specify a selected interval for each modification weight 112, and an interval index "d", and a modification weight 112 corresponding to a particular neuron 116 "W _{i, d, It} may include a neuron index "n" configured to identify as _n ". The set of indices may further include an access index "a" configured to count the number of times each modification weight 112 has been accessed by the input signal 104 during training of the p-net 100. Thus, the current status of each correction weight may adopt the name "W _{i, d, n, a} ".

  After frame 204, the method proceeds to frame 206. At frame 206, the method correlates one or more correction weights 112 from the plurality of correction weights disposed on the synapse 118 connected to the input 102 in question in correlation with the input value via the distributor 114. Including choosing. As mentioned above, each correction weight 112 is defined by its respective weight value. At frame 206, the method may further include assigning the plurality of influence factors 134 to the plurality of correction weights 112 via the distributor 114. At frame 206, the method may also include assigning each influence factor 134 to one of the plurality of correction weights 112 at a predetermined rate to generate a neuron sum 120. Also, at frame 206, the method may include summing, via the neuron 116, the product of the correction weight 112 and the assigned influence factor 134 across all synapses 118 connected thereto. In addition, at frame 206, the method is used to generate a neuron sum 120, part of the determined difference, according to the proportion established by the respective influence factor 134, via the weight correction calculator 122. It may include applying each correction weight 112.

As described above with respect to the structure of p-net 100, a plurality of influence factors 134 may be defined by the influence distribution function 136. In such a case, the method may further include receiving the input values within the range 137 of values divided into intervals "d" according to the interval distribution function 140, whereby each of the input values is , And each correction weight 112 corresponds to one of the intervals. The method also uses the input values received via the splitter 114 to select respective intervals "d", and the plurality of influence coefficients 134 correspond to the respective selected intervals "d". It may include assigning a correction weight 112, and at least one correction weight corresponding to an interval adjacent to the selected respective interval "d". As described above with respect to the structure of p-net 100, the correction weights 112 corresponding to the intervals adjacent to each selected interval "d" are, for example, W _{i, d + 1, n} or W _{i, d-1, n} May be identified as

  Following frame 206, the method proceeds to frame 208. At frame 208, the method includes summing the weight values of the selected modified weights 112 by the particular neuron 116 connected to the input 102 via the synapse 118 to generate a neuron sum 120. Each neuron 116 includes at least one output 117 as described above with respect to the structure of p-net 100. After frame 208, the method proceeds to frame 210. At frame 210, the method includes receiving, via weight correction calculator 122, a desired output signal 124 having signal values. Following frame 210, the method proceeds to frame 212. At frame 212, the method includes determining, via the weight correction calculator 122, the deviation 128 of the neuron sum 120 from the value of the desired output signal 124.

  As disclosed above in the description of p-net 100, the determination of the deviation 128 of the neuron sum 120 from the desired output signal value may include determining the mathematical difference between them. In addition, changing each of the correction weights 112 may include distributing mathematical differences to each of the correction weights used to generate the neuron sum 120. Alternatively, the distribution of the mathematical differences may include equally dividing the determined differences among each of the correction weights 112 used to generate the neuron sum 120. In yet another embodiment, determining the deviation 128 may also include dividing the desired output signal 124 value by the neuron sum 120, thereby generating a deviation factor. Further, in such a case, changing each of the correction weights 112 may include multiplying each correction weight 112 used to generate the neuron sum 120 by the generated deviation coefficient.

  After frame 212, the method proceeds to frame 214. At frame 214, the method includes, via the weight correction calculator 122, changing the respective correction weight values using the determined deviation 128. The modified correction weight values may then be summed or summed and then used to determine a new neuron sum 120. The summed modified modified weight values may then serve to train the p-net 100 by minimizing the deviation of the neuron sum 120 from the value of the desired output signal 124. Following frame 214, method 200 may include returning to frame 202 and performing additional training epochs until deviation of neuron sum 120 from the value of the desired output signal 124 is sufficiently minimized. In other words, additional training epochs can be performed to converge the neuron sum 120 to the desired output signal 124 to within the predetermined deviation or error value, whereby the p-net 100 is trained and new Make it possible to be considered ready for manipulation with

In general, the input image 106 needs to be prepared for p-net 100 training. Preparation of the p-net 100 for training generally begins with the formation of a set of training images, including the input image 106 and, in most cases, the desired output image 126 corresponding to the input image in question. The input image 106 (shown in FIG. 2) defined by the input signals I ₁ , I ₂ ... I _m for the training of the p-net 100 is a task which the p-net is tasked to process, eg human It is selected according to the recognition of images or other objects, recognition of specific activities, clustering or data classification, analysis of statistical data, pattern recognition, forecasting, or control of specific processes. Thus, the input image 106 may be in any format suitable for installation on a computer, for example, in the form of a table, chart, diagram and graphic, format, jpeg, gif or pptx, various document formats or symbols Can be presented using a set of

  The preparation for training of the p-net 100 also converts the selected input images 106 for their unification convenient for processing of the image in question by the p-net 100, eg all images In the case of the same number of signals or pictures, it may involve changing to a format with the same number of pixels. Color images could be presented, for example, as a combination of three basic colors. Image transformation also changes the characteristics of the image, for example moving the image in space, changing the visual characteristics of the image, such as resolution, brightness, contrast, color, viewpoint, perspective, focal length and focus, etc. And could include the addition of symbols, numbers, or notes.

  After selection of the number of intervals, a particular input image may be converted to an input image of interval format. That is, the actual signal value may be recorded as the number of the interval to which each signal in question belongs. This procedure may be performed at each training epoch for a given image. However, the image may also be formed once as a set of interval numbers. For example, in FIG. 7 the initial image is presented as a picture, while in the table "image in digital format" the same image is presented in the form of digital code and in the table "image in interval format" At this time, the image is presented as a set of interval numbers, where a separate interval is assigned for every 10 values of the digital code.

  The described structure of p-net 100 and the training algorithm or method 200 as described allow continuous or repetitive training of p-nets, and thus a complete set of training input images 106 at the start of the training process. There is no requirement to form It is possible to form a relatively small starting set of training images, and such starting set could be extended as needed. The input image 106 may be divided into different categories, eg, a set of pictures of one person, a set of pictures of a cat, or a set of pictures of a car, so that each category has a single output image , Corresponding to the name of such a person, or a specific label. The desired output image 126 represents a field or table of digital or analog values, each point corresponding to a particular numerical value between -∞ and + ∞. Each point of the desired output image 126 may correspond to the output of one of the p-net 100 neurons. The desired output image 126 may be encoded using a digital or analog code of the image, a table, text, a formula, a set of symbols such as barcodes, or audio.

  In the simplest case, each input image 106 may correspond to an output image encoded from the input image in question. One of such output image points may be assigned the maximum possible value, eg, 100%, whereas all other points may be assigned the minimum possible value, eg, 0. it can. In such a case, after training, probabilistic recognition of various images in the form of similarity rates with training images will be enabled. FIG. 8 shows a p-net 100 trained for the recognition of two images, a square and a circle, with a picture that includes some features of each shape, represented by a percentage whose sum is not necessarily equal to 100%. An example of what can be recognized as follows is shown. Such a process of pattern recognition by defining the similarity between different images used for training can be used to classify a particular image.

  To improve accuracy and eliminate errors, coding can be achieved with a set of several neural outputs rather than one output (see below). In the simplest case, the output image can be prepared prior to training. However, it is also possible for the output image to be formed by p-net 100 during training.

  In the p-net 100, the input image and the output image may be reversed. In other words, the input image 106 may be in the form of a field or table of digital values or analog values, where each point corresponds to one input of the p-net while the output image is , Presented in any format suitable for installation on a computer, for example, in the form of tables, charts, diagrams and graphics, jpeg, gif, pptx, various document formats, or a set of symbols It is also good. The obtained p-net 100 may be well suited for archiving systems as well as associative search of images, music expressions, formulas or data sets.

Following the preparation of the input image 106, typically, the p-net 100 needs to be formed and / or the parameters of the existing p-net have to be set to process a given task. The formation of p-net 100 may include the following designations.
The dimensions of p-net 100, as defined by the number of inputs and outputs
Synapse weights 108 for all inputs,
The number of correction weights 112,
The distribution of the modified weight influence factors (С _{i, d, n} ) for different values of the input signal 104, and the desired accuracy of training. The number of inputs is determined based on the size of the input image 106. For example, a large number of pixels may be used for a picture, while the number of outputs selected may depend on the size of the desired output image 126. In some cases, the number of outputs selected may depend on the number of training image categories.

  The values of the individual synapse weights 108 may be in the range of -∞ to + ∞. The value of synapse weights 108 less than 0 (0) may, for example, be derived from a particular input or for a more specific recognition of a human face in a picture containing a large number of different persons or objects. It may show signal amplification that may be used to enhance the effect of the signal from the image. On the other hand, values of synapse weights 108 greater than 0 (0) are used to indicate signal attenuation, which can be used to reduce the number of calculations required and to increase the speed of operation of the p-net 100. Can be In general, the higher the value of the synapse weight, the more the signal transmitted to the corresponding neuron is attenuated. If all synapse weights 108 corresponding to all inputs are equal and all neurons are equally connected to all inputs, then the neural network becomes universal and little is known in advance about the nature of the image Etc, it is most effective for common tasks. However, such a structure will generally increase the number of calculations required during training and operations.

  FIG. 9 shows an embodiment of p-net 100 in which the relationship between the input and each neuron is reduced according to a statistical normal distribution. An uneven distribution of synaptic weights 108 can result in the entire input signal being transmitted to a target or "central" neuron for a given input, thus assigning a value of 0 to the synaptic weights in question. . In addition, the non-uniform distribution of synaptic weights results in other neurons receiving reduced input signal values, eg, using normal, log normal, sine wave, or other distributions. Can bring. The value of synaptic weights 108 for neurons 116 receiving reduced input signal values may increase with increasing their distance from the "central" neurons. In such a case, the number of calculations can be reduced and the operation of the p-net can be accelerated. Such a network, which is a combination of a known fully connected neural network and an incompletely connected neural network, analyzes an image with strong internal patterns, for example a human face or a continuous frame of motion picture film It can be very effective for

  FIG. 9 illustrates one embodiment of a p-net 100 that is effective for local pattern recognition. To improve identification of common patterns, 10-20% strong connections where the value of synaptic weights 108 is small or 0, p in a deterministic or random approach, such as in the form of lattices, p It may be distributed throughout the net 100. The actual formation of the p-net 100 intended to process a specific task generates as a software object the main elements of the p-net, such as synapses, synapse weights, distributors, modification weights, neurons etc., eg It is accomplished using a program written in an object oriented programming language. Such programs may assign relationships between the objects described above and the algorithms that identify their actions. Specifically, synapse weights and correction weights may be formed at the beginning of the formation of p-net 100 with the setting of their initial values. The p-net 100 is completely formed before the start of its training, and it may be modified or extended in later frames, for example, when the information capacity of the network is exhausted, or when a fatal error occurs. It may be done. Completion of the p-net 100 is also possible while training is continuing.

  If p-net 100 is pre-formed, the number of correction weights selected on a particular synapse may be equal to the number of intervals within the range of the input signal. In addition, the correction weights may be generated as a signal in response to the appearance of the individual intervals after formation of the p-net 100. Similar to the classical neural network 10, parameter selection and p-net 100 settings are provided using a series of targeted experiments. Such experiments may include (1) formation of p-nets with the same synapse weights 108 at all inputs, and (2) evaluation of input signal values for the selected image and initial selection of the number of intervals. . For example, for the recognition of binary (one color) images, it may be sufficient to have only two intervals, and for qualitative recognition of 8-bit images, up to 256 intervals can be used. An approximation of a complex statistical dependency may require tens or even hundreds of intervals, and for large databases, the number of intervals could be thousands.

In the process of training p-net 100, the values of the input signals may be rounded as they are distributed during a particular interval. Therefore, an accuracy of the input signal greater than the width of the range divided by the number of intervals may not be required. For example, if the range of input values is set for 100 units and the number of intervals is 10, then an accuracy better than ± 5 will not be required. Such experiments may also include (3) selection of uniform distribution of intervals over the whole range of values of the input signal, the simplest distribution for the modified weight influence factor С _{i, d, n} The correction weights corresponding to the interval for the input signal of may be set equal to 1, while the correction weight impact for all remaining correction weights is set to 0 (0) Good. Such experiments may further include (4) training the p-net 100 with a predetermined accuracy using one, more, or all of the prepared training images.

The training time of p-net 100 for a given accuracy may be established by experimentation. If the accuracy and training time of the p-net 100 are satisfactory, it will be possible to maintain or change the selected settings. On the other hand, the search for more effective deformation is continued. If the required accuracy is not achieved, the impact of certain changes may be evaluated for optimization purposes. This may be performed one at a time or in groups. Such an assessment of the change may be changing the number of intervals, increasing or decreasing, changing the distribution type of the modified weight influence factor (係数_{i, d, n} ), normal distribution, power distribution, log Testing deformation with non-uniform distribution of intervals, such as using distribution or lognormal distribution, as well as changing the value of synaptic weights 108, eg, transitioning them to non-uniform distribution May be included.

  If the training time required for accurate results is considered to be excessive, training with an increased number of intervals may be evaluated for its effect on training time. As a result, if the training time is reduced, the number of intervals may be repeated until the desired training time is obtained without compromising the required accuracy. As the number of intervals is increased, the training time is not reduced, but if increased, the number of intervals may be reduced to perform additional training. If a reduction in the number of intervals results in a reduction in training time, the number of intervals could be further reduced until the desired training time is obtained.

  The formation of the p-net 100 settings may be through training using a predetermined training time and experimental determination of training accuracy. The parameters could be improved through experimental modifications similar to those described above. The practice of using various p-nets has shown that the procedure of setting selection is simple and does not require much time.

The actual training of p-net 100 as part of method 200, shown in FIG. 23, begins with providing input image signals I ₁ , I ₂ ... I _m to network input device 102. An input image signal is transmitted from the network input device 102 to the synapse 118, passes through the synapse weights 108, and enters a splitter (or group of splitters) 114. Based on the input signal values, the divider 114 sets the number of the interval "d" that a given input signal 104 corresponds to, and all of the weight correction blocks 110 of all synapses 118 connected to the respective input 102. Assign a correction weight influence factor С _{i, d, n} for the correction weight 112 of For example, if the interval "d" may be set to 3 for the first input, then С _{1,3, n} = 1 is set to 1 for all weights W _{1,3, n} , On the other hand, С _{i, d, n} may be set to 0 (0) for all other weights with i ≠ 1 and d ≠ 3.

For each neuron 116 identified as "n" in the relationship below, for every synapse 118 contributing to a particular neuron, corresponding to each correction weight 112 identified as W _{i, d, n} in the relationship below The neuron output sums 1 1, 重 み 2... _{N n} are formed by multiplying the corrected weight influence coefficients С _{i, d, n} and adding all the obtained values.
_{N n} = _{i i, d, n} Wi _{, d, n} × С _{i, d, n} [2]
The multiplication of W _{i, d, n} × С _{i, d, n} may be performed by various devices, eg, by distributor 114, a device with stored weights, or directly by neuron 116. The sum is forwarded to weight correction calculator 122 via neuron output 117. The desired output signals O ₁ , O ₂ ... O _n describing the desired output image 126 are also supplied to the calculator 122.

As described above, the weight correction calculator 122 calculates the modified values for the correction weights by comparing the neuron output sum Σ 1, 2 2... Σ _n with the desired output signals O ₁ , O ₂ . Is a computing device for FIG. 11 shows the set of correction weights W _{i, d, 1} that contribute to the neuron output sum _{1 1} multiplied by the corresponding correction weight influence factors C _{i, d, 1} and their product is then the neuron The output sum Σ1 is added.
Σ 1 = W 1, _{0, 1} × C 1, _{0, 1} . + W _1,1,1 × C _1,1,1 . + W _1,2,1 × C _1,2,1 . + ... [3]
When training starts, ie during the first epoch, the correction weights W _{i, d, 1} do not correspond to the input image 106 used for training, and hence the neuron output sum Σ 1 is , Not equal to the corresponding desired output image 126. Based on the initial correction weights W _{i, d, 1} , the weight correction system calculates a correction value Δ 1 which is used to change all correction weights that contribute to the neuron output sum Σ 1 (W _{i, d, 1} ) Do. The p-net 100 allows various options or variations for the p-net 100 to form and utilize a collective correction signal for all the correction weights W _{i, d, n} contributing to the specified neuron 116. Do.

In the following, two exemplary non-limiting variants for the formation and utilization of a collective correction signal are shown. Modification 1—Formation and utilization of a modified signal based on the difference between the desired output signal and the resulting output sum:
Calculation of equal correction value Δn for all correction weights contributing to neuron 'n' according to the following formula:
Δ _n = (O _n −Σ _n ) / S [4],
Where the desired output signal corresponding to On-neuron output sum Σ _n ,
S-Number of synapses connected to the neuron "n".
Change of all correction weights W _{i, d, n} contributing to the neuron "n" according to the following formula:
W _{i, d, n} modified = W _{i, d, n} + Δ _n / C _{i, d, n} [5]
Variant 2-Formation and use of a modified signal based on the ratio of the desired output signal to the obtained output sum as follows:
Calculation of equal correction value Δn for all correction weights contributing to neuron 'n' according to the following formula:
Δ _n = O _n / Σ _n [6],
Change of all correction weights W _{i, d, n} contributing to the neuron "n" according to the following formula:
Wi _{, d, n,} modified = Wi _{, d, n,} x? _N [7],
The modification of the correction weights W _{i, d, n} according to any available variant reduces the training error for each neuron 116 by focusing the output sum _{n n} of each neuron to the value of the desired output signal. Is intended. In this way, the training error for a given image can be reduced until it becomes equal to or approaches zero.

FIG. 11 shows an example of changing the correction weights _{Wi, d, n} during training. The values of the correction weights W _{i, d, n} are set in the form of a random weight distribution where the weight values are set to 0 ± 10% from the correction weight range before the training starts, and the final weight distribution is reached after training Do. The illustrated calculation of the collective signal is performed for all neurons 116 in p-net 100. The described training procedure for one training image can be repeated for all other training images. Such a procedure may cause the appearance of a training error on a portion of the previously trained image, as some correction weights _{Wi, d, n} may be involved in some images. Thus, training with another image may partially disturb the distribution of correction weights W _{i, d, n} formed for the previous image. However, due to the fact that each synapse 118 includes a set of correction weights W _{i, d, n} , training with new images may have caused p-net 100 to train previously, while possibly increasing the training error. Do not erase the Furthermore, the more synapse 118 contributes to each neuron 116 greatly, and the greater the number of correction weights W _{i, d, n} at each synapse _, the effect training for a particular image has on training for other images Is small.

  Each training epoch generally ends with substantial convergence of the overall training error and / or the local training error for all training images. Errors are evaluated using known statistical methods, such as, for example, Mean Squared Error (MSE), Mean Absolute Error (MAE), or Standard Error Mean (SEM). It can be done. If the total error, or part of the local error, is too high, additional training epochs may be performed until the error is reduced below a predetermined error value. The image recognition process described above (shown in FIG. 8), by defining the similarity rates between different images used for training, is in itself a classification of images along previously defined categories. It is a process.

  Clustering, ie, to divide the image into natural classes or groups not previously identified, the basic training algorithm of the method 200 takes a modified Self-Organizing Map (SOM) approach It can be used to change. The desired output image 126 corresponding to a given input image may be formed directly in the process of training p-net 100 based on the set of winner neurons having the maximum value of output neuron sum 120. FIG. 22 illustrates how the use of the basic algorithm of method 200 can generate a first-order set of output neuron sums. In this case, the set is further transformed such that some larger sums hold or increase their values, while all other sums are considered equal to zero. This transformed set of output neuron sums may be accepted as the desired output image 126.

  Formed as described above, the set of desired output images 126 includes clusters or groups. Thus, the set of desired output images 126 allows for clustering of non-linearly separable images. This is different from the classical network 10. FIG. 13 shows how the described approach can support clustering of complex virtual image "cat". In this case, different features of the image are assigned to different clusters-cats and cars. The set of desired output images 126 created as described may be used, for example, for creating different classifications, statistical analysis, image selection based on criteria formed as a result of clustering. Also, the desired output image 126 generated by the p-net 100 may be used as an input image for another or additional p-nets, which may also be formed according to the policies described for the p-net 100 in question. Can be Thus formed, the desired output image 126 may be used for subsequent layers of the multilayer p-net.

  The training of the classical neural network 10 is generally provided through a supervised training method based on a pre-prepared pair of input and desired output images. The same general method is also used for training p-net 100, but increasing the training speed of p-net 100 also allows training by external trainers. The role of the external trainer may, for example, be performed by a person or by a computer program. When acting as an external trainer, one may be involved in performing physical tasks or may operate within the gaming environment. The p-net 100 receives an input signal in the form of data regarding a particular situation and changes thereto. A signal reflecting the action of the trainer may be introduced as the desired output image 126, enabling the p-net 100 to be trained according to the basic algorithm. In this manner, modeling of various processes can be generated by p-net 100 in real time.

  For example, p-net 100 can be trained to drive a vehicle by receiving information regarding road conditions and driver actions. Through modeling of a wide variety of crisis situations, the same p-net 100 can be trained by many different drivers and accumulate more driving skills than is generally possible by any one driver. The p-net 100 has the ability to assess specific road conditions in 0.1 seconds or faster and build up a substantial "driving experience" that may enhance traffic safety in various situations. The p-net 100 may also be trained to cooperate with a computer, eg, a chess play machine. The ability of the p-net 100 to easily transition from training mode to recognition mode and vice versa allows the realization of a "learning from failure" mode when the p-net 100 is trained by an external trainer. In such a case, the partially trained p-net 100 may generate its own actions, for example, to control technical processes. The trainer will be able to control the actions of p-net 100 and modify those actions as needed. Thus, additional training of p-net 100 could be provided.

Although the information capacity of the p-net 100 is very large, it is not unlimited. Given a set dimension, such as the number of inputs, outputs, and intervals of p-net 100, if the number of images that p-net is trained to increase, then after a certain number of images, The number and magnitude of errors can also be increased. This way of error generation, as p-nets allow the number of neurons 116 and / or the number of signal intervals "d" to be increased between training epochs, across p-nets, or in their components. If such an increase is detected, the number and / or magnitude of the errors can be reduced by increasing the size of p-net 100. The expansion of p-net 100 adds new neurons 116, adds new inputs 102 and synapses 118, modifies the distribution of the correction weight influence factors C _{i, d, n} , and the existing intervals “ d "can be provided by splitting.

In most cases, p-net 100 will be trained to ensure its ability to recognize an image, pattern, and correlation specific to an image or set of images. The recognition process in the simplest case repeats the first steps of the training process according to the basic algorithm disclosed as part of the method 200. In particular,
Direct recognition starts with the formatting of the image according to the same rules used to format the training image,
The image is sent to the input of the trained p-net 100, the distributor assigns the correction weights W _{i, d, n} corresponding to the values of the input signal set during training, and the neurons are shown in FIG. Generate the respective neuron sum, as shown in
If the obtained output sum representing the output image 126 perfectly matches one of the p-net 100 trained images, the object has been recognized as snug
If the output image 126 partially matches some of the images for which the p-net 100 has been trained, the result indicates the percentage of agreement with different images. FIG. 13 shows that during recognition of complex images created based on combinations of cat and car images, the output image 126 represents a given combination of images and the percentage of each initial image's contribution to the combination It shows concretely how to instruct.

  For example, if several pictures of a particular person were used for training, the recognized image would be 90% for the first picture, 60% for the second picture, and the third picture. It can correspond to 35%. It is possible that the recognized image will, with some probability, correspond to the picture of another person or even an animal. This means that there is some similarity between the pictures. However, the probability of such similarity is usually lower. Based on such probabilities, the credibility of the recognition may be determined, for example, based on Bayesian theorem.

Using p-net 100, it is also possible to implement multi-step recognition combining the advantages of algorithmic and neural network recognition methods. Such multi-step recognition may include the following.
Initial recognition of the image by the network, which has been pretrained through using only 1% to 10% of the inputs, designated here as "basic inputs". Such portions of the input may be distributed in p-net 100 uniformly, randomly or with any other distribution function. For example, recognition of a person in a photo that includes multiple other objects,
Select the most informative object, or part of the object, for more detailed recognition. Such selection may be effected according to the structure of the particular object preset in memory, or according to the gradient, the brightness and / or the depth of the color of the image, as in the algorithmic method. For example, for recognition of portraits, the following recognition zones, eyes, mouth corners, nose shapes can be selected, and some specific features such as tattoos, car plate numbers, or house numbers can be selected and similar It can be recognized using an approach, and if necessary, a detailed recognition of the selected image is also possible.

  The formation of the computer emulation of p-net 100 and its training can be provided based on the above description by using any programming language. For example, object oriented programming may be used, where synapse weights 108, modification weights 112, distributors 114, and neurons 116 represent classes of programming objects or objects, and may be linked or linked between objects classes. Relationships are established and algorithms for interaction between objects and between object classes are set.

The formation and training of software emulation of p-net 100 may include the following.
1. Preparation of p-net 100 and preparation of training, specifically,
Converting a set of training input images into digital form, according to a given task
Analysis of the acquired digital image, including selection of parameters of the input signal to be used for training, eg frequency, magnitude, phase or coordinates, and Range of training signal, spacing within the range of the problem And the distribution of the correction weight influence factor 修正_{i, d, n} .
2. Software emulation of p-net including:
Formation of a set of inputs to p-net 100. For example, the number of inputs may be equal to the number of signals in the training input image
Formation of a set of neurons. Where each neuron represents a summing device,
Formation of a set of synapses with synaptic weights. Here, each synapse is connected to one p net input and one neuron,
Formation of weight correction blocks at each synapse. Here, the weight correction block includes a distributor and a correction weight, and each correction weight has the following characteristics.
○ Modified weight input index (i),
O Modified Weighted Neuron Index (n),
O Modified Weight Interval Index (d), and o Modified Weight Initial Value (W _{i, d, n} ).
• Specifying correlations between intervals and correction weights.
3. Training each neuron with one input image, including:
• Specification of the modified weight influence factor С _{i, d, n} which includes:
O determining the interval corresponding to the input signal of the training input image received by each input, and o specifying the magnitude of the modified weight influence factor С _{i, d, n} for all modified weights for all synapses.
・ Neurons for each neuron "n" by adding the correction weight values W _{i, d, n} of all synapse weights contributing to the neurons multiplied by the corresponding correction weight influence factor С _{i, d, n} Calculation of output sum (Σn).
_{N n} = _{i i, d, n} W _{i, d, n} × C _{i, d, n}
Calculation of the deviation or training error (T _n ) via subtraction of the neuron output sum _{n n} from the corresponding desired output signal O _n .
T _n = O _n −Σ _n ,
Calculation of equal correction values (Δ _n ) for all correction weights that contribute to neuron 'n' via dividing the training error by the number 'S' of synapses connected to neuron 'n'.
Δ _n = T _n / S
Change of all the correction weights W _{i, d, n} contributing to the respective neuron by adding the correction value Δ _n divided by the corresponding correction weight influence factor C _{i, d, n} to each correction weight.
Wi _{, d, n} modified = Wi _{, n, d} +? _N / Ci _{, d, n} .
Another method of calculating equal correction values (Δ _n ) and changing the correction weights W _{i, d, n} for all correction weights contributing to the neuron “n” may include:
Divide the signal O _n of the desired output image by the neuron output sum _{n n} .
Δ _n = O _n / Σ _n
Modification of the correction weights W _{i, n, d} that contribute to the neuron by multiplying the correction weights by the correction value Δ _n .
W _{i, d, n} modified = W _{i, d, n} × Δ _n
4. Training p-net 100 with all training images, including:
Repeating the process described above over all selected training images contained within one training epoch, and determining a set of errors or errors for a particular training epoch, with those errors at a predetermined tolerance level Compare and repeat training epochs until the training error is less than a predetermined tolerance level.

A practical example of software emulation of p-net 100 using object oriented programming is described below and is illustrated in FIGS.
The formation of the NeuronUnit object class may include the formation of:
A set of objects of the synapse class,
• The neurons 116 presenting variables, where the addition is performed during training, and the values of the desired neuron sum 120 are stored, and the calculation of the correction value Δ _n is performed during the training process, the variables are presented Calculator 122.
The class NeuronUnit for training p-net 100 may include the following.
・ Formation of neuron sum 120,
・ Setting of desired sum,
And correction value delta _n is calculated, and - modifying the weights _{W i, n,} calculated sum of the correction value delta _n to _d.
The formation of object class Synapse may include the following.
A set of correction weights W _{i, n, d} , and a pointer pointing to the input connected to the synapse 118.
Class Synapse may perform the following functions:
・ Initialization of correction weights W _{i, n, d}
The multiplication of the weight W _{i, n, d} by the factor С _{i, d, n} , and the correction of the weight W _{i, n, d} .
The formation of the object class InputSignal may include the following.
A set of indices on synapses 118 connected to a given input 102,
A variable containing the value of the input signal 104,
• possible minimum and maximum input signal values,
• The number of intervals “d”, and • the length of the intervals.
The class InputSignal may provide the following functionality:
-Formation of the structure of p-net 100, including the following.
O addition and deletion of links between the input 102 and the synapse 118, and o setting of the number of intervals "d" for the synapse 118 of a particular input 102.
Setting the parameters of the minimum and maximum input signal 104,
Contribution to the operation of the p-net 100.
設定 setting of the input signal 104, and 設定 setting of the correction weight influence factor С _{i, d, n} .
The formation of object class PNet includes the following set of object classes.
-NeuronUnit, and-InputSignal.
The class PNet provides the following functions:
・ Setting the number of objects of InputSignal class,
-Setting of the number of objects of the NeuronUnit class, and-Group request of the functions of the objects NeuronUnit and InputSignal.
During the training process, cycles can be formed,
Before the cycle starts, a neuron output sum equal to 0 is formed,
All synapses contributing to a given NeuronUnit are reviewed. Every synapse 118
○ Based on the input signal 102, the distributor forms a set of modified weight influence factors С _{i, d, n} ,
○ All weights W _{i, n, d of the} synapse 118 are reviewed, and for each weight
■ The values of the weights W _{i, n, d} are multiplied by the corresponding modified weight influence factors С _{i, d, n} ,
■ The result of the multiplication is added to the forming neuron output sum,
The correction value Δ _n is calculated,
The correction value Δ _n is divided by the correction weight influence factor С _{i, d, n} , ie, Δ _n / С _{i, d, n} , and • all synapses 118 contributing to a given NeuronUnit are reviewed. For each synapse 118, all weights W _{i, n, d} of the synapse in question are reviewed, and for each weight, its value is changed to the corresponding correction value Δ _n .

The above mentioned possibilities of the additional training of the p-net 100 allow a combination of training and image recognition, which allows the training process to be speeded up and its accuracy to be improved. When training p-net 100 with a set of sequentially changing images, such as training with successive frames of film slightly different from one another, the additional training may include:
・ Training using the first image,
Recognition of the next image and identification of the similarity between the new image and the one the network was initially trained on. If the recognition error is less than the predetermined value, no additional training is required, and if the recognition error exceeds the predetermined value, additional training is performed.

Training of the p-net 100 with the basic training algorithm described above is effective to solve the problem of image recognition but does not eliminate data loss or corruption due to duplicate images. Thus, the use of p-net 100 for storage purposes, although possible, can not be completely reliable. The present embodiment describes training of p-net 100 that provides protection against loss or corruption of information. An additional restriction may be introduced into the basic network training algorithm that requires that any correction weights W _{i, n, d} can be trained only once. After the first training cycle, the values of the weights W _{i, n, d} remain fixed or constant. This may be achieved by including an additional access index "a" for each correction weight, which is the above-mentioned index representing the number of accesses to the correction weight W _{i, n, d} of the problem during the training process.

As mentioned above, each correction weight may take on the names Wi _{, n, d, a} . Here, "a" is the number of accesses to the problem weights during the training process. In the simplest case, for weights that have not been modified or fixed, a = 0, while for weights modified or modified by the above-described basic algorithm, a = 1. is there. Furthermore, while applying the basic algorithm, the modified weights W _{i, n, d, a} with fixed value a = 1 can be excluded from the weights to be corrected. In such a case, equations [5], [6], and [7] can be modified as follows.

The above limitations may be partially applied to the correction of the previously trained correction weights _{Wi, n, d, a} , but may only be applied to the weights forming the most important image. For example, within training with a set of single person portraits, one particular image may be declared primary and given priority. After training with such a preferred image, all the modified weights W _{i, n, d, a} modified in the training process may be fixed, ie index a = 1, hence the weights Designated as _{Wi, n, d, 1} other images of the same person may remain modifiable. Such preferences may include other images, such as those used as encryption keys and / or contain significant numerical data.

The change of the correction weights W _{i, n, d, a} may also be limited to the increase of the index "a", rather than being completely forbidden. That is, the subsequent use of each of the weights _{Wi, n, d, a} may be used to reduce its ability to change. The more frequently _a particular correction weight W _{i, n, d, a} is used, the smaller is the change in weight per access, and thus, the previous stored during training with subsequent images The image is less changed and the damage incurred is reduced. For example, when a = 0, any change of the weights W _{i, n, d, a} is possible, and when a = 1, the possibility of changing the weight is ± 50% of the value of the weight In the case of a = 2, the probability of change may be reduced to ± 25% of the weight value.

After reaching the predetermined access number, as indicated by the index "a", further changes in the weights _{Wi, n, d, a} may be prohibited, for example when a = 5. Such an approach may provide a combination of high intelligence and information security in a single p-net 100. The network error calculation mechanism can be used to set the level of tolerance so that information with losses within a predetermined accuracy range can be stored. Here, the accuracy range may be assigned according to a particular task. In other words, for a p-net 100 operating with a visual image, the error may be set to a level that can not be caught by the naked eye, which causes a significant increase in storage capacity. Will be brought. The above may enable the generation of visual information, for example, a very efficient storage of movies.

The ability to selectively clean computer memory may be beneficial for the continued high level functionality of p-net 100. Such selective cleaning of the memory can be performed by deleting certain images without loss or corruption of the remaining part of the stored information. Such cleaning may be effected as follows.
• Identification of all the correction weights W _{i, n, d, a} involved in the image formation, eg by introducing an image into the network or putting together a list of correction weights used per image
The reduction of the index "a" for each correction weight W _{i, n, d, a} , and the correction weight W _{i, n, d, a} reduced to 0, or the index "a" to 0 When replacing with one of the random values close to the middle of the range of possible values for the problem weight.

The appropriate order and succession of the reductions of the index "a" can be selected experimentally to identify strong patterns hidden within the sequence of images. For example, every time 100 images are introduced into p-net 100 during training, index "a" may be reduced by a count of 1 until "a" reaches a value of 0. In such cases, the value of "a" may be increased in response to the introduction of a new image. The competition between the increase and decrease of the 'a' is that random changes are gradually removed from the memory, while being used many times, the modified correction weights _{Wi, n, d, a} stored It can lead to situations that can be When the p-net 100 is trained, for example, using multiple images with similar attributes in the same problem environment or similar environments, the correction weights W _{i, n, d, a} often used are their values Constantly confirm, the information in these areas becomes very stable. Furthermore, random noise will gradually disappear. In other words, the p-net 100 can play the role of an effective noise filter with the decreasing index "a".

  The above described embodiment of p-net 100 training without loss of information makes it possible to create p-net memories with high capacity and reliability. Such memory can be used as a large capacity high-speed computer memory that delivers even greater speed than a "cache memory" system, but with the cost and complexity of the computer being typical of a "cache memory" system. There is no increase. According to published data, generally, while recording a movie using a neural network, the memory can be compressed by tens or hundreds of times without significant loss of recording quality. In other words, the neural network can operate as a very effective archive program. Combining this ability of neural networks with the fast training capabilities of p-net 100 may allow the creation of a fast data transmission system, a memory with a high storage capacity, and a fast decoding program multimedia file, i.e. codecs.

In the p-net 100, because of the fact that the data is stored as a set of code records, as _a set of correction weights _{Wi, n, d, a} , via the existing method and using the same network Do not consider decryption or unauthorized access to p-net. Therefore, p-net 100 can provide a significant degree of data protection. Also, unlike conventional computer memories, damage to the individual storage elements of p-net 100 has less adverse effects as it substantially compensates for the function of the other elements being lost. In the image recognition process, the intrinsic pattern of the image being used is practically undistorted as a result of the damage of one or more elements. The above can dramatically improve computer reliability and allow the use of specific memory blocks that would be considered defects under normal conditions. In addition, this type of memory is less vulnerable to hacker attacks since there is no permanent address for the important bytes in p-net 100, and the memory is attacked by such systems with various computer viruses. It becomes difficult to receive

The above described process of image recognition using the determination of similarity rates between different images used in training can also be utilized as a process of image classification according to previously defined categories as described above. The basic training process can be modified for clustering, which is the division of the image into non-predefined natural classes or groups. This embodiment may include the following.
・ Preparation of a set of training input images, not including prepared output images
Network formation and training using the formation of neuron output sums as performed according to the basic algorithm,
In the obtained output image, the output with the largest output sum, ie the selection of a group of winner outputs, which may be organized like a winner output or a Kohonen network
Creation of the desired output image, in which the winner output or the group of winner outputs receives the maximum value. at the same time,
O The number of winner outputs to be selected may be predetermined, eg, in the range of 1 to 10, or the winner outputs may be selected according to the rule "N% or more of the maximum neuron sum""N" may be, for example, within 90-100%,
O All other outputs may be set equal to zero.
• training according to the basic algorithm by using the desired output image created, Fig. 13, and · repeating the whole procedure for the other images with the image-by-image formation of different winners or winner groups.

  The set of desired output images formed in the manner described above may be used to describe clusters or groups into which multiple input images may be naturally separated. Such a set of desired output images can be used to create different classifications, such as for image selection in statistical analysis, according to established criteria. The above may also be used for the above described reversal of input and output images. In other words, the desired output image may be used as an input image for another or additional network, and the additional network output is presented in any form suitable for computer input. Image may be used.

  In p-net 100, after a single cycle of training using the above algorithm, the desired output image can be generated with some variation of the output sum. This may reduce the speed of the training process and may also reduce its accuracy. In order to improve the training of the p-net 100, the variation of the point size will cover the full range of possible output values, eg -50 to +50, as shown in FIG. The initial variation of points may be artificially increased or expanded. Such an extension of the initial variation of points may be either linear or non-linear.

  Situations may arise in which the maximum value of a particular output is an outlier or an error, for example the appearance of noise. This can be manifested by the appearance of maxima surrounded by a large number of small signals. When the winner output is selected, small signal values can be ignored through selection as the largest signal winner surrounded by other large signals. Known statistical techniques of variance reduction, such as importance sampling, may be used for this purpose. Such an approach may allow noise removal while maintaining the basic useful pattern. The creation of the winner group allows clustering of the linearly inseparable images, ie images associated with more than one cluster, as shown in FIG. The above results in a significant improvement in accuracy and can reduce the number of clustering errors.

  In the process of training p-net 100, typical errors that are subject to correction are:

  Error correction is also possible with the aid of the above mentioned algorithm in training with an external trainer.

  The hardware portion of p-net 100 may be provided in the form of digital, analog, or complex digital-analog microchips. A representative p-net 100 microchip can be utilized for both information storage and processing. The microchip of p-net 100 may be based on various variable resistors, field effect transistors, memristors, capacitors, switching elements, voltage generators, non-linear photovoltaic cells, etc. Variable resistors may be used as synaptic weights 108 and / or correction weights 112. A plurality of such resistors may be connected in parallel, in series or in series-parallel. In the case of parallel connection of the respective resistors, the signal can be encoded by the current value, so that an automatic analog summation of the currents can be facilitated. Two sets of resistors, excitatory and inhibitory, may be provided on each synapse to obtain positive or negative signals. In such a hardware structure, the inhibitory signal can be subtracted from the excitatory signal.

  Each correction weight 112 may be implemented as a memristor-like device. As understood by those skilled in the art, a memristor is a variable resistor having a resistance controlled by the current in the circuit or by the potential or charge. Appropriate memristor functionality may be achieved via the actual memristor device, its software or physical emulation. During operation of p-net 100 at low voltage potentials, the memristor can operate as a simple resistor. During the training mode, the resistance of the memristor can be changed, for example by a strong voltage pulse. How the change in value of the memristor (increase or decrease in resistance) may depend on the polarity of the voltage, while the magnitude of the change in value may depend on the magnitude of the voltage pulse .

  FIG. 24 illustrates one embodiment of a p-net 100 labeled p-net 100A that is similar in all respects to the p-net 100 described above, except having the specific elements that will be described later. In p-net 100A, each correction weight 112 is established by a memory element 150 that holds the respective weight value of a particular correction weight. In p-net 100A, weight correction calculator 122 may be configured to change the respective correction weight value of correction weights 112 established by the corresponding memory element 150. Consistent with other embodiments of p-net 100A, correction weights 112 are established in corresponding memory elements 150 using the determined deviation 128. During operation of p-net 100A shown in FIG. 25, the respective outputs 117 of each of all neurons 116 provide respective neuron sums 120 and establish the operation output signal 152 of p-net 100A. The motion output signal 152 has signal values and represents either a portion or all of the motion output image 154.

  During training of p-net 100A, weight correction calculator 122 receives the desired output signal 124 representing a portion or all of output image 126, and the deviation 128 of neuron sum 120 from the value of the desired output signal 124. The determined and determined deviations may be used to change the respective correction weight value established by the corresponding memory element. Further, summing the modified correction weight values of the correction weights 112 established by the corresponding memory element 150 and determining the neuron sum minimizes the deviation of the neuron sum 120 from the desired output signal value 124. It will be. The p-net 100A is trained with minimizing the departure of the neuron sum 120 from the desired output signal value 124.

  As shown in phantom in FIG. 24, the trained p-net 100A of any of the disclosed embodiments uses only the supplemental input signal 156 having a value with the corresponding supplemental desired output signal 158. Can be configured to receive supplemental training. In other words, the previously trained p-net 100A is complemented without retaining some or all of the original input signal 104 and the desired output signal 124 utilized to initially train the p-net 100A. I can receive training. Each of the plurality of synapses 118 in p-net 100A may be configured to receive one or more additional modification weights 112 established by the respective memory element 150. Such additional correction weights 112 may be added to the synapse either during training of the p-net 100A or prior to supplemental training. Such additional correction weights 112 may be used to extend the number of memory elements 150 available to train and operate the p-net 100A.

  The p-net 100A may also be configured to remove one or more modification weights 112 established by the respective memory element 150 from the respective synapse 118 either during or after the training of the p-net 100A. . Removal of some of the correction weights 112 may allow the neural network to hold only the number of memory elements needed to operate the neural network. Such ability to remove the correction weights 112 is intended to make the p-net more compact and hence more effective for training and subsequent operation. The p-net 100A also receives an additional input 102, an additional neuron 116 with each additional neuron output 117 and an additional synapse 118 either before or during training of the p-net, thereby p It may be configured to extend the operating parameters of the net. Such additions to p-net 100A may enhance capabilities such as capacity, output accuracy, and the number of tasks that can be processed by p-net.

  The p-net 100A further includes any number of unused inputs 102, neurons 116, each additional neuron output 117, and either before, during, or after the p-net's initial training or supplemental training. It may be configured to delete along with the synapse 118. Such an ability to eliminate elements of unused p-net 100A simplifies the structure and changes the operating parameters of p-net without loss of output quality of p-net, ie compresses p-net It is intended to do.

  As shown in FIG. 26, each memory element 150 is configured to define a weight value of a respective correction weight 112, such as resistance, impedance, capacitance, magnetic field, induction, or electric field strength. And / or may be established by an electrical component or device 160 characterized by magnetic properties. Such an electrical device 160 may, for example, be a memristor (shown in FIGS. 26-28), a resistor (shown in FIGS. 29-32), a transistor, a capacitor (shown in FIGS. 29-32), an electric field. It can be configured as an effect transistor, a photoresistor or light dependent resistor (LDR), a magnetic dependent resistor (MDR), or a thermistor. As understood by those skilled in the art, a thermistor is a resistor having a memory capable of performing logic operations and storing information, generally a three terminal implementation of memristor.

  In such embodiments of memory element 150, the respective electrical and / or magnetic properties of each electrical device 160 may be configured to be changed during training of p-net 100A. In addition, in the p-net 100A using the embodiment of the electrical device 160 of the memory element 150, the weight correction calculator 122 alters the electrical and / or magnetic properties of each of the corresponding devices utilized by the p-net 100A. Thus, the value of each correction weight 112 may be changed. Each electrical device 160 is also modified during training of the p-net 100A, as described above, and the electrical characteristics and / or corresponding to the values of the respective correction weights 112 used during the operation of the p-net after training. It may be configured to maintain or maintain magnetic properties.

  Particular embodiments of suitable memristor may be both a known physical representation of the device, as well as a functional representation or equivalent of the software or electrical circuitry of the device. Figures 26-28 illustrate embodiments of a representative p-net 100A that utilizes such physical memristor. As shown in FIG. 26, each input 102 is configured to receive an analog input signal 104 and may be from an external source, such as an image sensor, an array of light sensitive elements or microphones, a digital to analog converter, etc. The input signals are represented as voltages V1, V2 ... Vm. All input signals 104 combine to generally describe the corresponding input image 106.

  Each memory element 150 may also be established by a block 162 having an electrical resistor 164. Such a block 162 having an electrical resistor 164 may include a selection device 166. The selection device 166 selects one or more electrical resistors 164 from the block 162 using the determined deviation 128 of the neuron sum 120 from the value of the desired output signal 124 as described above, each correction weight It is configured to establish 112. Additionally, each memory element 150 established by the block 162 with the electrical resistor 164 may also include an electrical capacitor 168. In other words, each memory element 150 may be established by block 162 having an electrical resistor 164 and an electrical capacitor 168. In such case, the selection device 166 may be further configured to select the capacitors 168 and the electrical resistors 164 using the determined deviation 128 to establish each correction weight 112.

  Each of the embodiments of p-net 100A shown in FIGS. 24 and 25 may be configured as any of analog, digital, and digital-analog neural networks. In such an embodiment of p-net 100A, of the plurality of inputs 102, the plurality of synapses 118, the memory element 150, the set of distributors 114, the set of neurons 116, the weight correction calculator 122, and the desired output signal 124. Any of may be configured to operate in analog format, digital format, and digital-analog format. FIG. 26 shows the p-net 100A in the first phase of training ending with the determination of the deviation 128 of the neuron sum 120 from the value of the desired output signal 124, while FIG. 27 shows in the second phase of training The p-net 100A is shown terminating with the formation of a correction signal 170 for the correction weights 112 established by the electrical device 160. As shown in FIG. 26, each synapse 118 includes a plurality of electrical devices 160, shown as memristor, connected to one of the plurality of inputs 102 and functioning as a correction weight 112. Each correction weight 112 is defined by the value of the electrical resistance.

  The p-net 100 A shown in FIG. 26 also includes a set of distributors 114. Each distributor 114 receives its respective input signal 104 via a set of electrical devices 160 configured as a memristor, operating on one of a plurality of inputs 102 for establishing the appropriate correction weights 112. It is connected possible. In addition, each distributor 114 is configured to select one or more correction weights 112 embodied by the memristor from the plurality of available correction weights in correlation to the input voltage. FIG. 28 shows a p-net 100A using an electrical device 160 configured as a memristor arranged in a dual parallel branch. With respect to the other solutions for the construction of p-net 100A described above, FIGS. 29-31 show electrical device 160 configured as a common resistor to define the appropriate resistance in p-net 100A, On the other hand, FIG. 32 shows an electrical device 160 configured to define the impedance in p-net 100A.

In p-net 100A configured as an analog or analog-to-digital network, each input 102 is configured to receive an analog or digital input signal 104. Here, in FIG. 24 and FIG. 25, input signals are represented as I ₁ , I ₂ ... I _m . Each input signal I ₁ , I ₂ ... I _m represents some analog characteristic of the input image 106, such as, for example, values such as magnitude, frequency, phase, signal declination, and the like. In the analog p-net 100A, each input signal 104 has an input analog value, and collectively, the plurality of input signals 104 generally describe the analog input image 106. In p-net 100A configured as an analog network, each neuron 116 may be established by either serial or parallel communication channel 172, eg, a wire, or a serial or parallel bus. The exemplary bus embodiment of communication channel 172 may use both parallel and bit-serial connections, as will be appreciated by those skilled in the art. For example, if the corresponding analog signal is provided via current, the communication channel 172 may be a serial current bus while the corresponding analog signal is via the potential on the respective correction weight 112. If provided, the exemplary communication channel may be a parallel bus.

26-32, an analog embodiment of the correction weights 112 is shown. Each analog correction weight 112 is defined by memory element 150. Memory element 150 holds a respective weight value of a particular modified weight, and flows forward through memory element 150, a forward signal (from input to output)
Or reverse signal (from output to input)
Or act on additional control signals to change them. Furthermore, each analog correction weight 112 may hold its respective weight value during actual operation of p-net 100A, eg, during image recognition. In addition, the value of each analog correction weight 112 is a forward signal during training of p-net 100A.
Or reverse signal
Or may be changed via additional control signals. As a result, the analog embodiment of p-net 100A may allow for the generation of microchips that enable support for various computer applications. Furthermore, the entire p-net 100A may be programmed into an electronic device having a memory, such as a microchip, a video card or the like. Thus, a suitable embodiment of each memory element 150 will now be stored in the memory of the electronic device in question as well.

  Consistent with the above, each electrical device 160 in the analog p-net 100A restores the electrical and / or magnetic properties corresponding to the modified value of the respective correction weight 112 after p-net training. Can be configured as follows. The weight correction calculator 122 may be configured to generate one or more correction signals that represent the determined deviation 128 of the neuron sum 120 from the value of the desired output signal 124. Additionally, each of the generated correction signals may be used to alter the electrical and / or magnetic properties of the at least one electrical device 160. That is, a separate correction signal is used for each device to be changed. Weight correction calculator 122 may also be configured to generate a single correction signal that is used to change the electrical and / or magnetic properties of each electrical device 160. That is, one correction signal may be used for all electrical devices to be changed.

  The particular embodiment of the weight correction calculator 122 may be incorporated into the p-net 100A as programmed software or be established via an external device or accessible computer program. For example, weight correction calculator 122 may be established as a set of differential amplifiers 174. Consistent with the entire disclosure, each such differential amplifier 174 is configured to generate a respective correction signal that represents the determined deviation 128 of the neuron sum 120 from the value of the desired output signal 124. obtain. Each electrical device 160 may be configured to maintain electrical and / or magnetic characteristics corresponding to the modified values of the respective correction weights 112 after training of the p-net 100A is completed. The p-net 100A may also be configured to use the maintained electrical and / or magnetic properties during operation of the p-net, ie, after training has been completed. Such a structure of p-net 100A facilitates parallel or batch training of correction weights 112, thereby the amount of time required to train p-net as compared to the classical neural network described above. Enable a significant reduction of

  Each distributor 114 in p-net 100A takes a single input signal 104 and selects one or more of a plurality of data output lines connected to a single input, an analog device, a digital device, Or, it can be configured as either an analog-to-digital device, ie as a demultiplexer 176. Such demultiplexer 176 may be configured to select one or more of the correction weights 112 from the plurality of correction weights in response to the received input signal 104. Each distributor 114 may be configured to convert the received input signal 104 into a binary code and to correlate to the binary code to select one or more correction weights 112 from the plurality of correction weights.

  FIG. 33 illustrates a method 300 of operating a utility neural network 100B (shown in FIG. 25). Method 300 operates in accordance with the above-described disclosure of FIGS. 2-22 and 24-32. Method 300 begins at frame 302. At frame 302, the method includes providing a utility neural network 100B. Following frame 302, the method proceeds to frame 304. At frame 304, the method processes the data via the utility neural network 100B with the modified values of the correction weights 112 established during its training by a separate similar neural network, such as p-net 100A. To do. The training of the similar p-net 100A may include supplemental training, as described above with reference to FIG.

  The utility neural network 100B and the trained separate p-net 100A can be made similar, for example by having a matching neural network structure, as represented in FIG. 25, whereby the utility neural network 100B may eliminate weight correction calculator 122 and the corresponding ability to train a utility neural network. Thus, the matching neural network structure may include the same number of inputs 102, correction weights 112, distributors 114, neurons 116, neuron outputs 117, and synapses 118. In other words, p-net 100A and utility neural network 100B may be substantially identical for all the features that establish the operating parameters of the network. The largest functional difference is that p-net 100A can be trained by establishing modified correction weights 112. Each of the analog p-net 100A and the utility neural network 100B can be implemented in different configurations, for example, various forms of hardware and / or software, and analog and / or digital formats, such that the utility neural network and Similar p-nets can be represented by heterogeneous carriers. In such cases, a translation mechanism (not shown) may be utilized to transform or interpret the data having the modified correction weights 112.

Each correction weight 112 may be established by the memory element 150 in each of the utility neural network 100B and the trained separate p-net 100A. Specifically, in p-net 100A, memory element 150 may hold respective modified weight values corresponding to modification weights 112 after p-net training. Each input image 106 provided to the utility neural network 100B may be represented by a combined input signal 104 represented by I ₁ , I ₂ ... I _m , as described with respect to FIG. As described additionally with respect to FIG. 2, each input signal I ₁ , I ₂ ... I _m represents the value of some characteristic of the corresponding input image 106.

  From frame 304, the method proceeds to frame 306. At frame 306, the method includes establishing the motion output signal 152 with the modified correction weights 112 as described with respect to FIG. Thus, processing of input data, such as input image 106 via utility neural network 100B, may be concluded as an operation output signal 152 of the utility neural network with such data recognition. The operation output signal 152 of the utility neural network 100B may then be interpreted or decoded either by the utility network itself or by an operator of the utility network as part of a frame 308 to complete the method. The establishment of the modified correction weights 112 in the p-net 100A may be accomplished according to the method 200 of training the p-net 100, as described with respect to FIG.

  34 and 35 illustrate an embodiment of p-net 100 labeled p-net 100 B, which is similar in all respects to p-net 100 described above, except having particular elements to be described later. Show. Furthermore, the p-net 100B shown in FIGS. 34 and 35 operates with the alignment structure, ie is configured to be trained for subsequent recognition of other images using the selected image ing. The term "image" as used herein is intended to indicate any kind of information or data received for processing or generated by a neural network. In FIG. 36, the trained p-net 100B is designated via a symbol 100C. When p-net 100B is being trained, input image 106 is defined as a training image, while in trained p-net 100C, input image 106 is intended to be recognized. The training image 106 is received as a training input value array 107 by a plurality of inputs 102, or is organized as a training input value array 107 during training of the p-net 100B, ie after being received by a plurality of inputs. Either.

  Similar to the other embodiments, each of p-net 100B and p-net 100C shown in FIGS. 34-36 also includes synapses 118, each synapse 118 being connected to one of a plurality of inputs 102. And may include multiple correction weights 112 and may also include synaptic weights 108. The correction weights 112 for all synapses 118 are organized as a correction weight array 119A, ie, in the form of a correction weight array 119A. Thus, in FIGS. 34-36, the correction weight array 119A includes all the correction weights 112 within the dashed box 119A. p-net 100 B may also include a set of distributors 114. In such an embodiment, each splitter 114 is operatively connected to one of a plurality of inputs 102 for receiving a respective input signal 104. The embodiment of p-net 100B, including modified weight array 119A, may also be characterized by the absence of different neuron units 119 while retaining some of the components of the neuron units.

  Similar to p-net 100 and p-net 100A described above, p-net 100B further includes a set of neurons 116, providing a means for performing the actions described in detail below. Each neuron 116 also has at least one output 117 and is connected to at least one of the plurality of inputs 102 via one synapse 118. Each neuron 116 similarly sums the values of the correction weights 112 selected from each synapse 118 connected to the respective neuron 116, thereby generating and outputting the neuron sum array 120A separately noted as n n. Is configured as. In this embodiment, as shown in FIGS. 34-36, separate distributors 114 may be used for each synapse 118 of a given input 102 as well. Alternatively, a single splitter may be used for all such synapses (not shown). During formation or setup of the p-net 100B, all the correction weights 112 are assigned initial values that may change during the process of p-net training, as shown in FIG. The initial value of the correction weights 112 may be randomly selected, calculated with the aid of a predetermined mathematical function, selected from a predetermined template, and so on. The initial value of the correction weights 112 may be the same or different for each correction weight 112 and may be zero.

  As shown in FIGS. 34 and 35, p-net 100B also includes a controller 122A configured to direct the training of p-net 100B, and therefore to perform the actions described in detail below. It becomes a means of The controller 122A may include the weight correction calculator 122 described above for the other embodiments. The controller 122A includes memory to properly perform the tasks described in detail below. At least a portion of the memory is tangible and non-transitory. The memory of controller 122A may be a recordable medium involved in providing computer readable data or processing instructions. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media for controller 122A may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random access memory (DRAM), which may constitute the main memory. Such instructions may be transmitted by one or more transmission media including coaxial cables, copper wire and fiber optics, including electrical wires including a system bus coupled to the processor of the computer.

  The memory of controller 122A may also include suitable media, such as magnetic or optical media. The controller 122A may include a high speed clock, required analog-to-digital (A / D) and / or digital-to-analog (D / A) circuitry, necessary input / output circuitry and devices (I / O), and appropriate signals. It may be configured or equipped with other required computer hardware, such as adjustment and / or buffer circuitry. The algorithms required by controller 122A or accessible by controller 122A may be stored in memory and automatically implemented to provide the necessary functionality described in detail below.

  Controller 122A may be programmed to organize correction weights 112 into correction weight array 119A. The controller 122A is also configured to receive a desired image or output signal 124 organized as a desired output value array 126A. The controller 122A is further configured to determine a deviation 128 of the neuron sum array 120A from the desired output value array and generate a deviation array 132. The controller 122A is further configured to change the correction weight array 119A using the determined deviation array 132. In such a case, summing the modified correction weight values to determine neuron sum array 120A reduces, ie, minimizes, the deviation 128 of neuron sum array 120A from the desired output value array 126A. , Generate a trained correction weight array 134A (shown in FIG. 36). Similar to the correction weight array 119A shown in FIGS. 34 and 35, the trained correction weight array 134A includes all the correction weights 112 in the dashed box 134A. Also, as shown in FIG. 36 and similar to the correction weight array 119A in FIGS. 34 and 35, the trained correction weight array 134A includes all the trained correction weights 112A in the dashed box 119A. , A distributor 114 associated with them. Thus, the minimized deviation 128 of the neuron sum array 120A compensates for the error generated by the p-net 100B. In addition, the generated trained correction weight array 134A facilitates simultaneous or parallel training of the p-net 100B.

  In trained p-net 100C, shown in FIG. 35, multiple inputs 102 to the p-net may be configured to receive input image 106. Such an input image 106 may either be received as an input value array 107A or may be organized as an input value array 107A during image recognition by p-net 100B. Each synapse 118 may include a plurality of trained correction weights 112A. In addition, each neuron 116 may be configured to sum the weight values of the trained correction weights 112A corresponding to each synapse 118 connected to the respective neuron, whereby the plurality of neurons is A recognized image array 136 is generated, which results in the recognition of the input image 106. In the embodiment of p-net 100B and trained p-net 100C including a set of distributors 114, the distributors organize the training and input images 106 as respective training input value arrays 107 and input value arrays 107A. Can be configured. Thus, such a set of distributors 114 is operatively connected to a plurality of inputs 102 for receiving each of the respective training and input images 106. The above-described operations may be performed using a structured matrix, specifically, a trained modified weight matrix, instead of the trained modified weight array 134A, as described in detail below.

  The controller 122A can be further programmed with the target deviation array or target deviation array 138 of the neuron sum array 120A from the desired output value array 126A. Further, controller 122A may be configured to complete training of p-net 100B when deviation 128 of neuron sum array 120A from desired output value array 126A falls within tolerance 139 of target deviation array 138. The tolerance range 139 may be checked against the maximum value or the minimum value in the target deviation array 138, or the average value of the target deviation array 138. Alternatively, the controller 122A is configured to complete training of the p-net 100B when the reduction of the deviation 128 or the speed of convergence of the training input value array 107 and the desired output value array 126A falls to the predetermined speed value 140. It can be done. Acceptable range 139 and / or predetermined velocity value 140 may be programmed into controller 122A.

  Training input value array 107, input value array 107A, correction weight array 119A, neuron sum array 120A, desired output value array 126A, deviation array 132, trained correction weight array 134A, recognized image array 136, and target deviation The array 138, ie, the parameter values therein, are trained in the training input value matrix 141, the input value matrix 141A, the correction weight matrix 142, the neuron sum matrix 143, the desired output value matrix 144, the deviation matrix 145, respectively. It may be organized as a modified weight matrix 146, a recognized image matrix 147, and a target deviation matrix 148. In this case, in each respective array 107, 107A, 119, 120, 126, 132, 134, 136 and 138, the values of the respective parameters are organized, for example, in the form of a data table accessible to the processor The values in each of the matrices 141, 141A, 142, 143, 144, 145, 146, 147, and 148 may in particular apply algebraic matrix operations individually to each respective matrix, and Organized to allow application to combinations. Matrices 141, 141A, 142, 143, 144, 145, 146, 147, and 148 are not specifically shown in the figure, but when so organized, the respective arrays 107, 107A, 119, 120, It should be understood to replace 126, 132, 134, 136 and 138.

  In the following example, a specific matrix with any number of columns and rows is shown for illustrative purposes. For example, training images may be received and / or organized in the form of an input training matrix | I |.

  The training input image matrix described above may then be converted to a training input value matrix 141, represented as matrix | C |, via controller 122A. Each matrix | C | will have a corresponding number of columns for the number of inputs “I”, but taking into account a certain number of intervals “i”, and a corresponding number of rows for the number of images. .

  In the matrix | C |, intervals “i” identified with the specific correction weights 112 to be used during training. Within the column corresponding to interval "i", the value of the signal may be replaced with 1 (1) to indicate that a particular signal is to be used at a particular interval, while At other intervals for the signal in question, the value of the signal may be replaced with 0 (0) to indicate that a particular interval will not be considered.

  An exemplary modified weighting matrix 146 may be formed as the matrix | W | shown below.

The neuron sum matrix 143 may be represented as the matrix | Σ | shown below.

  The desired output value matrix 144 may be formed as matrix | O |, as shown below.

The departure 128 of the neuron sum matrix 143 can be determined from the desired output value matrix 144 to generate a departure matrix 148, which is represented as the matrix | E |
here,
Etc.

The modified weighting matrix 142, which will be represented as matrix | W | in the following, may be modified using the determined deviation matrix 145. This sums the values of the modified correction weights 112, determines the neuron sum matrix 143, minimizes the deviation of the neuron sum matrix 143 from the desired output value matrix 144, and is represented as the matrix | W _trained | It is possible to generate a trained correction weight matrix 146. The matrix | W _trained | is derived according to the equation | W _trained | = | W | + | ∇W | (where the factor | ∇W | will be described in detail below).

  As discussed above, the formation of the trained correction weight array 134A and the trained correction weight matrix 146 facilitates simultaneous training of the p-net 100B.

  In the image recognition embodiment (shown in FIG. 36) using the trained p-net 100C, simultaneous recognition of batches of the input image 106 may be effected using the matrix operations described above. Specifically, a modified p-net 100C, a modified weight array that may be represented as a two-dimensional n x k matrix | W |. Here, “n” is the number of neurons 116 and “k” is the number of correction weights 112 in a particular neuron. The matrix | W | can be expressed generally as:

  For simultaneous recognition of batches of input image 106, the input image to be recognized may be presented as v x k matrix | Ir |. Here, “v” is the number of recognizable images and “k” is the number of correction weights 112 in a particular neuron 116. The matrix | Ir | of the input image 106 for recognition may be generally represented as:

  In the above matrix | Ir |, each row of the matrix is a single image to be recognized.

The simultaneous recognition of a batch of input images 106 multiplies the matrix | W | by the transposed matrix | I | ^T and is represented by the symbol “| Y |” and is represented by: It can be brought about by generating.
| Y | = | W | × | Ir | ^T
The matrix | Y | has dimensions n x v. Each column of the matrix | Y | is a single output or recognized image obtained by the trained p-net 100C. The matrix | Y | can be generally indicated as follows.

  Each of p-nets 100B and 100C may further include data processor 150, which may be a subunit of controller 122A. In such an embodiment, controller 122A further includes at least one of respective training input value matrix 141, input value matrix 141A, modified weight matrix 142, neuron sum matrix 143, and desired output value matrix 144. It may be configured to partition or divide into each submatrix. The controller 122A may also be configured to communicate the plurality of obtained submatrices or submatrices to the data processor 150 for separate mathematical operations with it. Such partitioning of each of the problem matrices 141, 142, 143, and 144 into respective submatrices is simultaneous or parallel data processing, as well as image recognition of the input value matrix 141A or training of the p-net 100B. To facilitate any increase in speed. Such simultaneous or parallel data processing also enables the scalability of p-net 100B or 100C. That is, by changing the size of the p-net by limiting the size of each matrix undergoing algebraic operations on a particular processor and / or splitting the matrix among multiple processors, such as the illustrated processor 150 Bring the ability. As shown in FIGS. 34-36, in such an embodiment of p-nets 100B and 100C, multiple data processors 150 in communication with controller 122A may be part of controller 122A or may be deployed at the end. Regardless of whether they are, they may be configured to be utilized, operate separately and in parallel.

The controller 122A may modify the modified weighting matrix 142 by applying algebraic matrix operations to the training input value matrix 141A and the modified weighting matrix to train the p-net 100B. Such mathematical matrix operations may include determining the mathematical product of the input value matrix 141A and the modified weight matrix 146, thereby forming the weight matrix 151 of the current training epoch. The controller 122A may also be configured to subtract the neuron sum matrix 143 from the desired output value matrix 144 to generate a matrix 153 of neuron sum deviations shown as the matrix | E | described above. In addition, the controller 122 A divides the neuron sum deviation matrix 153 by the number of synapses 118 connected to each neuron 116, identified below with the letter “m”, such as In addition, it may be configured to generate a matrix 155 of deviations for each neuron input, represented by the symbol “| ΔW |” below.
| ΔW | = | E | / m

The controller 122A may be further configured to determine the number of times each correction weight 112 has been used during one training epoch of the p-net 100B, represented by the symbol "| S |" As shown below, the matrix | S | is obtained through multiplying the training input value matrix 141A by a unit vector.
The controller 122A further uses the determined number of times each correction weight was used during one training epoch to generate an average, hereinafter represented by the symbol “| ∇W |”, for one training epoch. It may be configured to form a deviation matrix 157.
| ∇W | = | ΔW | / | S |
Further, controller 122A adds the mean deviation matrix 157 for one training epoch to the modified weight matrix 142, thereby generating a _trained corrected weight matrix 146, denoted in the following as | W _trained | It may be configured to complete one training epoch as shown below.
| W _trained | = | W | + | ∇W |

  FIG. 37 illustrates a method 400 for operating p-net 100B as described above with reference to FIGS. 34-36. Method 400 is configured to improve the operation of a device, such as a computer or computer system, utilized in performing supervised training with one or more data processors, such as processor 150. There is. Method 400 may be programmed in a non-transitory computer readable storage device encoded with instructions executable to perform the method for operating p-net 100B.

  Method 400 begins at frame 402. At frame 402, the method includes receiving a training image 106 via the plurality of inputs 102. As described above with respect to the structure of p-net 100B shown in FIGS. 34 and 35, training image 106 may be received as training input value array 107 prior to the start of the training phase in question, or of the actual training phase. May be either organized as a training input value sequence. Following frame 402, the method proceeds to frame 404. At frame 404, the method includes organizing the correction weights 112 of the plurality of synapses 118 into a correction weight array 119A. Each synapse 118 is connected to one of the plurality of inputs 102 and includes a plurality of correction weights 112, as described above with respect to the structure of p-net 100B.

  After frame 404, the method proceeds to frame 406. At frame 406, the method includes generating a neuron sum sequence 120A via the plurality of neurons 116. Each neuron 116 has at least one output 117 and is connected to at least one of the plurality of inputs 102 via one of the plurality of synapses 118, as described above with respect to the structure of p-net 100B. There is. Furthermore, each neuron 116 is configured to sum the weight values of the correction weights 112 corresponding to each synapse 118 connected to the respective neuron. Following frame 406, at frame 408, the method includes receiving, via controller 122A, the desired image 124 organized as the desired output value array 126A. After frame 408, the method proceeds to frame 410. At frame 410, the method includes, via the controller 122A, determining the deviation 128 of the neuron sum array 120A from the desired output value array 126A, thereby generating the deviation array 132.

  Following frame 410, the method proceeds to frame 412. At frame 412, the method includes, via the controller 122A, modifying the correction weight array 119A with the determined deviation array 132. The modified correction weight values of the modified correction weight array 119A may then be summed or summed and then used to determine a new neuron sum array 120A. The summed modified modified weight values of the modified modified weight array 119A then reduce or minimize the deviation of the neuron sum array 120A from the desired output value array 126A, and the trained modified weight array 134A It can serve to generate. Deviation array 132 is sufficiently minimized when deviation 128 of neuron sum array 120A from desired output value array 126A falls within tolerance 139 of target deviation array 138, as described above for the structure of p-net 100C. It may be determined to be The trained correction weight array 134A includes the trained correction weights 112A determined using the deviance array 132, thereby training the p-net 100B.

  As described above with respect to the structure of p-net 100B, training input value array 107, correction weight array 119A, neuron sum array 120A, desired output value array 126A, deviation array 132, trained correction weight array 134A, and target deviation Each of the arrays 138 is organized as a training input value matrix 141, a correction weight matrix 142, a neuron sum matrix 143, a desired output value matrix 144, a deviation matrix 145, a trained correction weight matrix 146, and a target deviation matrix 148, respectively. It can be done. At frame 412, the method causes at least one of the respective training input value matrix 141, input value matrix 141A, modified weight matrix 142, neuron sum matrix 143, and desired output value matrix 144 via controller 122A. It may further include partitioning into each submatrix. The submatrices thus obtained may be conveyed to the data processor 150 for separate mathematical operations with it, which may facilitate simultaneous data processing and an increase in the speed of training of the p-net 100B. .

  At frame 412, the method also modifies the modified weighting matrix 142 by applying algebraic matrix operations to the training input value matrix 141 and the modified weighting matrix via the controller 122A, thereby training the p-net 100B. Can be included. Such mathematical matrix operations may include determining the mathematical product of the training input value matrix 141 and the modified weight matrix 142, thereby forming the weight matrix 151 of the current training epoch. At frame 412, the method may further include, via controller 122A, subtracting the neuron sum matrix 143 from the desired output value matrix 144 to generate a matrix 153 of neuron sum deviations. Also, at frame 412, the method divides, via controller 122A, the matrix 153 of neuron sum deviations by the number of inputs connected to each neuron 116 to generate a matrix 155 of deviations for each neuron input May include.

  Further, at frame 412, the method may include determining, via controller 122A, the number of times each correction weight 112 has been used during one training epoch of p-net 100B. Also, the method further comprises, via the controller 122A, forming an average deviation matrix 157 for one training epoch using the determined number of times each correction weight 112 was used during a particular training epoch. May be included. For example, such an operation divides, for each element, the matrix of deviations per neuron input by the determined number of times each correction weight was used during a particular training epoch, and during one training epoch The average deviation for each correction weight 112 used may be obtained, which may include forming an average deviation matrix 157 for one training epoch.

  In addition, in frame 412, other matrix-based operations are used to form mean deviation matrix 157 for one training epoch, eg using arithmetic means, geometric means, harmonic means, root mean square, etc. May be Also, at frame 412, the method adds, via controller 122A, the mean deviation matrix 157 for one training epoch to the modified weight matrix 142, thereby generating a trained corrected weight matrix 146, and identifying May include completing a training epoch. Thus, by allowing matrix operations to be applied in parallel to all the correction weights 112, the method 400 can simultaneously generate p-nets 100B in training, and so on. As a result, speed is improved, facilitating training.

  Following frame 412, method 400 may include returning to frame 402 and performing additional training epochs until deviation array 132 is sufficiently minimized. In other words, additional training epochs can be performed to converge the neuron sum array 120A to the desired output value array 126A to within a predetermined deviation or error value, whereby the p-net 100B is trained. , Be prepared to be ready for manipulation with the new input image 106. Thus, after frame 412, the method may proceed to frame 414 for image recognition with trained p-net 100C (shown in FIG. 36).

  In the image recognition embodiment using trained p-net 100 C, at frame 414, method 400 includes receiving input image 106 via multiple inputs 102. As described above with respect to the structure of p-net 100C, either input image 106 is received as input value array 107A or is organized as an input value array during recognition of the image by p-net 100C. It can be. Continuing to frame 414, at frame 416, the method includes assigning the plurality of trained correction weights 112A of the trained correction weight array 134A to each synapse 118. After frame 416, the method proceeds to frame 418.

  At frame 418, the method includes summing the weight values of the trained correction weights 112A corresponding to each synapse 118 connected to the respective neuron 116. Such a summation of the weight values of the trained correction weights 112A allows the plurality of neurons 116 to generate a recognized image array 136, as described above with respect to the structure of p-net 100B. , Provide recognition of the input image 106. In addition to the matrices 141, 142, 143, 144, 145, 146 and 148 used for training, as described above for the structure of p-net 100C, the input value array 107A and the recognized image array 136 are: Each may be organized as an input value matrix 141A and a recognized image matrix 147.

  At frame 418, the method may also include partitioning any of the used matrices, such as the input value matrix 141A, into respective submatrices via the controller 122A. The sub-matrices thus obtained are conveyed to the data processor 150 for separate mathematical operations using it, thereby facilitating simultaneous data processing and increased speed of image recognition of the p-net 100C. obtain. Similar to the effect that matrix operations have on the training portion of method 400 in frames 202-212, if algebraic matrix operations are applied in parallel to the matrix or submatrix of trained p-net 100C, then in frames 214-218. The image recognition part benefits from the speed improvement. Thus, by allowing matrix operations to be applied in parallel to all trained correction weights 112A, the method 400 simultaneously and consequently with speed improvement using p-net 100C, Make image recognition easy. Following frame 418, the method proceeds to frame 402 for additional training, as described with respect to FIGS. 34-36, if the image recognition achieved is deemed to lack sufficient accuracy. And the method may end at frame 420.

  The detailed description and drawings or figures support and explain the present disclosure, but the scope of the present disclosure is defined only by the claims. Although some of the best mode and other embodiments for implementing the claimed disclosure have been described in detail, various alternatives for practicing the disclosure as defined in the appended claims Various designs and embodiments exist. Moreover, the features of the embodiments shown in the drawings, or of the various embodiments described in the present description, are not necessarily to be understood as embodiments independent of one another. Rather, each of the features described in one of the example embodiments may be combined with one or more other desired features from other embodiments, verbally or with reference to the drawings. It is possible to create other embodiments which will not be described. Accordingly, such other embodiments are included within the scope of the appended claims.

Claims (24)

A neural network implemented by a computer,
A plurality of inputs to the neural network configured to receive a training image, the training image being received as a training input value array, or the training input values during training of the neural network Multiple inputs, either organized as an array,
A plurality of synapses, each synapse being connected to one of the plurality of inputs, comprising a plurality of correction weights, each correction weight being defined by a weight value, the correction weights of the plurality of synapses Are organized in a modified weight array, with multiple synapses,
A plurality of neurons, each neuron having at least one output and being connected to at least one of the plurality of inputs via at least one of the plurality of synapses, each neuron being A plurality of neurons, configured to add the weight values of the correction weights corresponding to the synapses connected to the respective neurons, whereby the plurality of neurons generate a neuron sum array;
A controller,
Receiving a desired image organized as a desired output value array;
Determining a deviation of the neuron sum array from the desired output value array, and generating a deviation array;
Modifying the respective modified weight values in the modified weight array using the determined deviation array, thereby summing the modified corrected weight values and determining the neuron sum array To reduce the deviation of the neuron sum array from the desired output value array, generate a trained correction weight array, and thereby facilitate simultaneous training of the neural network. When,
With the controller, configured to
Equipped with
The training input value array, the correction weight array, the neuron sum array, the desired output value array, the deviation array, and the trained correction weight array are a training input value matrix, a correction weight matrix, and a neuron sum, respectively. Organized as a matrix, a desired output value matrix, a deviation matrix, and a trained correction weight matrix,
neural network. In a trained neural network,
The plurality of inputs to the neural network are configured to receive an input image, wherein the input image is received as an input value matrix, or the recognition is performed during recognition of the image by the neural network. Either organized as an input value matrix,
Each synapse includes a plurality of trained correction weights of the trained correction weight matrix,
Each neuron is configured to sum the weight values of the trained correction weights corresponding to each synapse connected to the respective neuron, whereby the plurality of neurons are identified by the recognized image Generate a matrix, which results in the recognition of the input image,
The neural network according to claim 1.   The apparatus further comprising a set of distributors, wherein the set of distributors is configured to organize each of the training image and the input image as the respective training input value matrix and input value matrix, and the distributor The neural network of claim 2, wherein a set is operatively connected to the plurality of inputs for receiving the respective training image and the input image.   The controller is further programmed with a matrix of target deviations of the neuron sum matrix from the desired output value matrix, and the controller further comprises the deviation of the neuron sum matrix from the desired output value matrix. 3. The neural network of claim 2, wherein the neural network is configured to complete training of the neural network when it is within the tolerance of the matrix of target deviations. (Cancel)   The controller further comprises a plurality of data processors, and the controller further includes at least one of the respective input value matrix, the training input value matrix, the modified weight matrix, the neuron sum matrix, and the desired output value matrix. Are partitioned into respective submatrices, and a plurality of said obtained submatrices are communicated to said plurality of data processors for separate parallel mathematical operations using them, whereby simultaneous data processing as well as said inputs 5. The neural network of claim 4, configured to facilitate increasing the speed of one of image recognition of a value matrix and training of the neural network.   5. The neural network of claim 4, wherein the controller modifies the modified weighting matrix by applying algebraic matrix operations to the training input value matrix and the modified weighting matrix, thereby training the neural network.   The neural network according to claim 7, wherein the mathematical matrix operation comprises determining a mathematical product of the training input value matrix and the modified weighting matrix, thereby forming a weighting matrix of a current training epoch. . The controller
Subtracting the neuron sum matrix from the desired output value matrix to generate a matrix of neuron sum deviations;
Dividing the matrix of neuron sum deviations by the number of inputs connected to the respective neuron to generate a matrix of deviations for each neuron input;
9. The neural network of claim 8, further configured to: The controller
Determining the number of times each correction weight has been used during one training epoch of the neural network;
Forming an average deviation matrix for the one training epoch using the determined number of times each correction weight was used during the one training epoch;
Adding the mean deviation matrix for the one training epoch to the modified weight matrix, thereby generating the trained modified weight matrix and completing the one training epoch;
10. The neural network of claim 9, further configured to: A method for operating a computer implemented neural network, comprising:
Receiving a training image via a plurality of inputs to the neural network, the training image being received as a training input value matrix, or the training input value matrix during training of the neural network Receiving, either being organized as
Organizing the correction weights of the plurality of synapses into a correction weight matrix, each synapse being connected to one of the plurality of inputs, comprising a plurality of correction weights, each correction weight being a weight value Prescribed, organized and
Generating a neuron sum matrix via a plurality of neurons, each neuron having at least one output, and at least one of the plurality of inputs via one of the plurality of synapses Generating, each connected being configured to sum up the weight values of the correction weights corresponding to each synapse connected to the respective neuron;
Receiving, via a controller, a desired image organized as a desired output value matrix;
Determining the deviation of the neuron sum matrix from the desired output value matrix via the controller to generate a deviation matrix;
Changing the respective modified weight values in the modified weight matrix using the determined deviation matrix via the controller, thereby summing the modified corrected weight values, the neuron Determining a sum matrix causes the deviation of the neuron sum matrix from the desired output value matrix to be reduced, producing a trained correction weight matrix, thereby facilitating simultaneous training of the neural network Change, and
Method, including. In a trained neural network,
Receiving an input image via the plurality of inputs to the neural network, the input image being received as an input value matrix, or the input during recognition of the image by the neural network Receive, which is either organized as a value matrix
Attributing a plurality of trained correction weights of the trained correction weight matrix to each synapse, wherein each trained correction weight is defined by a weight value,
The weights of the trained correction weights corresponding to each synapse connected to the respective neuron are summed, whereby the plurality of neurons generate a recognized image matrix, thereby the input Bring recognition of the image,
The method of claim 11.   The method further comprises: organizing each of the training image and the input image as the respective training input value matrix and input value matrix via a set of distributors, the set of distributors comprising: 13. The method of claim 12, operatively connected to the plurality of inputs for receiving an input image.   The controller is further programmed with a matrix of target deviations of the neuron sum matrix from the desired output value matrix, and the method, via the controller, the neuron sums from the desired output value matrix 13. The method of claim 12, further comprising: completing training of the neural network when the deviation of a matrix falls within the tolerance of the matrix of target deviations. (Cancel)   The neural network further includes a plurality of data processors, and the method further comprises, via the controller, the respective input value matrix, the training input value matrix, the modified weight matrix, the neuron sum matrix, and the desired output. At least one of the value matrices is partitioned into respective submatrices, and the plurality of obtained submatrices are communicated to the plurality of data processors for separate parallel mathematical operations using them, 15. The method according to claim 14, further comprising facilitating the speed of one of simultaneous data processing and image recognition of the input value matrix and training of the neural network.   The method of claim 14 further comprising modifying the modified weighting matrix by applying algebraic matrix operations to the training input value matrix and the modified weighting matrix via the controller, thereby training the neural network. The method described in.   18. The method according to claim 17, wherein applying the mathematical matrix operation comprises determining a mathematical product of the training input value matrix and the modified weight matrix, thereby forming a weight matrix of a current training epoch. Method described. Subtracting the neuron sum matrix from the desired output value matrix via the controller to generate a matrix of neuron sum deviations;
Dividing the matrix of neuron sum deviations by the number of inputs connected to the respective neuron via the controller to generate a matrix of deviations for each neuron input;
The method of claim 18, further comprising: Determining, via the controller, the number of times each correction weight has been used during one training epoch of the neural network;
Forming an average deviation matrix for the one training epoch using the determined number of times each correction weight was used during the one training epoch via the controller;
Adding the mean deviation matrix for the one training epoch to the modified weighting matrix via the controller, thereby generating the trained modified weighting matrix and completing the one training epoch ,
20. The method of claim 19, further comprising A non-transitory computer readable storage device for operating an artificial neural network, said storage device comprising
Receiving a training image via a plurality of inputs to the neural network, the training image being received as a training input value matrix, or the training input value matrix during training of the neural network Receiving, either being organized as
Organizing the correction weights of the plurality of synapses into a correction weight matrix, each synapse being connected to one of the plurality of inputs, comprising a plurality of correction weights, each correction weight being a weight value Prescribed, organized and
Generating a neuron sum matrix via a plurality of neurons, each neuron having at least one output, and at least one of the plurality of inputs via one of the plurality of synapses Generating, each connected being configured to sum up the weight values of the correction weights corresponding to each synapse connected to the respective neuron;
Receiving a desired image organized as a desired output value matrix;
Determining the deviation of the neuron sum matrix from the desired output value matrix, and generating a deviation matrix;
Changing the respective modified weight values in the modified weight matrix using the determined deviation matrix, thereby summing the modified corrected weight values and determining the neuron sum matrix To reduce the deviation of the neuron sum matrix from the desired output value matrix, generate a trained correction weight matrix, thereby facilitating simultaneous training of the neural network When,
A storage device in which executable instructions are encoded to do Receiving an input image via the plurality of inputs to the neural network, the input image being received as an input value matrix, or the input during recognition of the image by the neural network Receiving, which is either systematized as a value matrix,
Attributing a plurality of trained correction weights of the trained correction weight matrix to each synapse, each trained correction weight being defined by a weight value;
The weights of the trained correction weights corresponding to each synapse connected to the respective neuron are summed, whereby the plurality of neurons generate a recognized image matrix, thereby the input Bring recognition of the image,
22. The storage device of claim 21, wherein the executable instructions are further encoded to: An apparatus for operating an artificial neural network, comprising
Means for receiving a training image via a plurality of inputs to the neural network, the training image being received as a training input value matrix, or the training input during training of the neural network Means, which are either systematized as a value matrix,
Means for organizing a plurality of synapse modification weights into a modification weight matrix, each synapse being connected to one of the plurality of inputs, including a plurality of modification weights, each modification weight being a weight Means, defined by the values,
Means for generating a neuron sum matrix via a plurality of neurons, each neuron having at least one output and at least one of the plurality of inputs via one of the plurality of synapses Means, connected with each other, each neuron being configured to sum the weight values of the correction weights corresponding to each synapse connected to the respective neuron,
Means for receiving a desired image organized as a desired output value matrix;
Means for determining the deviation of the neuron sum matrix from the desired output value matrix and generating a deviation matrix;
Means for changing the respective modified weight values in the modified weight matrix using the determined deviation matrix, thereby summing the modified corrected weight values and determining the neuron sum matrix Means for reducing the deviation of the neuron sum matrix from the desired output value matrix, producing a trained correction weight matrix, thereby facilitating simultaneous training of the neural network. When,
An apparatus comprising: In a trained neural network,
Means for receiving an input image via the plurality of inputs to the neural network, wherein the input image is received as an input value matrix or during recognition of the image by the neural network Means, which are either organized as said input value matrix,
Means for assigning a plurality of trained correction weights of said trained correction weight matrix to each synapse, wherein each trained correction weight is defined by a weight value,
Means for summing the weights of the trained correction weights corresponding to each synapse connected to the respective neuron, whereby the plurality of neurons generate a recognized image matrix, Means for providing recognition of the input image by
24. The apparatus of claim 23, comprising: