KR20180070187A - Apparatus and method for regularizating of neural network device - Google Patents

Abstract

The present technique relates to a neuromorphic apparatus capable of normalizing a computing operation by performing dropout on some information, and an operation method thereof. The neuromorphic apparatus includes: an input unit for applying a plurality of input signals to a plurality of corresponding first lines, respectively; an operation unit including a plurality of memory elements cross-connected between the first lines and a plurality of second lines, in which each of the memory elements has a unique weight value so as to generate product signals of the input signals of the corresponding first lines and weights to output the generated product signals to corresponding second lines among the second lines; a drop-connect control unit including switches connected between the first line and the memory elements to drop a connection of the input signal applied to a corresponding memory element by switching the switch through a random signal; and an output unit connected to the second line to activate signals of the second lines to apply the activated signals to the input unit as an input signal and to output a result when a preset number of operations are completed.

Description

[0001] APPARATUS AND METHOD FOR REGULARIZING OF NEURAL NETWORK DEVICE [0002]

The present invention relates to an apparatus and method for performing normalization of a neural network device using a drop connect and / or a dropout.

There are hundreds of billions of neurons in the human brain, which form a complex neural network. Neurons are able to exert their intellectual abilities by sending and receiving signals through thousands of other neurons and synapses. Neurons are structural and functional units of the nervous system and can be the basic unit of information transmission. Synapses can be junctions between neurons and synaptic contact with other neurons. The neural network device can be a device that makes the artificial neurons simulating these neural networks to the neuron level.

The neural network device may be a method of arbitrarily allocating specific information to a neuron and learning allocation information to a corresponding neuron. In a neural network device, a neuron can be in contact with the synapse of another neuron to input information, and can also output the input information of another neuron.

The algorithm of the neural network device may be highly complex. For example, one neuron can input a lot of information. And can perform an operation of calculating the input information with the respective weights. Therefore, in order to improve the complexity of the neural network device, normalization (regularization) is required.

Embodiments of the present invention may provide an apparatus and method for normalizing an operation of a neural network device.

Embodiments of the present invention may provide an apparatus and method for drop-connecting a signal input to a memristor in a neural network device that processes information using memristors.

Embodiments of the present invention may provide an apparatus and method that can drop out node outputs of a memristor in a neural network device that processes information using memristors.

Embodiments of the present invention may provide an apparatus and method that can be normalized using a drop-connect and drop-out function in a neural network device that processes information using memristors.

A neural network apparatus according to various embodiments of the present invention includes an input unit for applying a plurality of input signals to a plurality of first lines respectively corresponding to the plurality of first lines and a plurality of second lines, Wherein each of the memory devices has a unique weight value and generates product signals of the input signal and the weight of the corresponding first line to generate corresponding second And a switch connected between the first line and the memory elements, the switch being controlled by the switch to switch the input signal to a corresponding memory element, A connect control unit and the second line, activates signals of the second line and applies the input signals to the input unit, When the output operation is ended output that may include a.

The neural network apparatus according to various embodiments of the present invention may further include an input unit for applying a plurality of input signals to a plurality of first lines respectively corresponding to the plurality of first lines and a plurality of second lines, Wherein each of the memory elements has a unique weight value and generates product signals of the input signal and the weight of the corresponding first line to output a corresponding product of the plurality of second lines, And a switch connected between the first line and the memory elements, wherein the switch is controlled by a first random signal to switch a connection of an input signal applied to the corresponding memory element, And a switch connected to the second lines, wherein the second random signal causes at least one of the second lines A dropout control unit for dropping out a signal and an output unit connected to the second line and activating signals of the second line to apply the input signals to the input unit and outputting the signals when the set number of operations is completed . .

A method of normalizing a neural network device according to various embodiments of the present invention includes the steps of: selecting a plurality of memory devices, each having a specific resistance value corresponding to a weight value, which are located at intersections of a plurality of first lines and a plurality of second lines, Applying a plurality of input signals to the first lines and a corresponding one of the first switches among the first switches connected between the first lines and the memory elements by a first random signal, And a memory circuit for storing the current signals generated by the input signals and the resistance values corresponding to the memory elements in the second lines, And an output step of activating the signals of the second lines by an activation function and feeding back the signals of the second lines to the input signal .

Some of the information may be dropped out in a neural network device based on a memory element (e.g., a memristor) to normalize and / or refine the computational operation, in accordance with embodiments of the present invention.

1 is a diagram showing a structure of a neuron.
2 is a diagram showing a Perceptron structure.
3 is a diagram showing a multi-layer perceptron (MLP) structure.
4A and 4B are views for explaining a normalization method in a neural network apparatus according to various embodiments of the present invention.
5 is a diagram illustrating a configuration of an apparatus for implementing a drop connect function in a neural network apparatus using memristors according to various embodiments of the present invention.
6 is a diagram showing a configuration of a neural network device according to various embodiments of the present invention.
7 is a diagram showing a configuration of a neural network device according to various embodiments of the present invention.
8A to 8D are views for explaining the operation of the output unit in the neural network apparatus according to various embodiments of the present invention.
9 is a diagram for explaining a loop operation in which an output signal is fed back to a next layer in a neural network device according to various embodiments of the present invention.
10A and 10B are diagrams showing a configuration of a random signal generator in a neural network apparatus according to various embodiments of the present invention.
11 is a diagram showing a configuration of another embodiment for performing a normalization operation in a neural network apparatus according to various embodiments of the present invention.
12 is a diagram showing a configuration of a neural network device according to various embodiments of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, only parts necessary for understanding the operation according to the present invention will be described, and the description of other parts will be omitted so as not to disturb the gist of the present invention.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a diagram showing a structure of a neuron.

Referring to FIG. 1, a neural network algorithm can be an algorithm that is created by mathematically modeling a mammalian brain. The mammalian brain is made up of a number of monolithic entangled entities, which can be neurons with the same structure as in Fig. In a neuron, synapse 101 can deliver an electrical signal to the synapse of another neuron. The manner in which the synapse 101 transmits an electrical signal may be N: N matching rather than 1: 1 matching. That is, one synapse can carry an electrical signal to one synapse, or one synapse can transmit an electrical signal to multiple synapses.

FIG. 2 is a diagram showing a Perceptron structure, and FIG. 3 is a diagram showing a multi-layer perceptron (MLP) structure.

Referring to FIG. 2, a perceptron may be a mathematical model that imitates a neuron unit, and MLP may be a structure in which a perceptron is networked. Perceptron and MLP can be basic models of neural network circuits. Perceptron can be an algorithm that makes human neurons, neurons, into a computable form.

The perceptron may multiply the plurality of inputs and the weights corresponding to each of the inputs, respectively, and then generate a summed product of the multiplication signals. 2 shows an example of a structure for multiplying three inputs (x0-x2) and corresponding three weights w0-w2, respectively, and adding up product signals through an adder (sum of products, vector multiplication) have. In FIG. 2, b denotes a bias and can be used to learn a threshold value for determining whether to activate the input data. Perceptron can be an algorithm that can solve the linear detachable (detachable with weight sum) problem. The perceptron having the structure as shown in FIG. 2A can perform the linear separation function, but can not perform the nonlinear separation function.

Perceptrons can find linear boundaries for linear classification of learning vectors into two classes. The weight may be a value indicating the direction or shape of the linear boundary. The bias may be a value representing an intercept of a linear boundary, and the threshold may mean a minimum value for which a value is activated. The activation function is defined as the net value (sum of product of input and weight, SOP, which can be geometrically interpreted as an equation of linear boundary) If it is greater than 1, it outputs 1, and if it is smaller than the threshold, it is 0. Other types of active functions can be used in MLP. That is, the neuron is the smallest unit constituting the neuron network. If the SOP is larger than the threshold value, the neuron outputs 1 while being activated. When the SOP is smaller than the threshold, 0 is output. A single layer Perceptron as shown in FIG. 2 may be composed of an input layer and an output layer. The input layer may be a layer into which a learning vector or an input vector is input. The data of the input layer can be transferred to the output layer neuron and output as a value according to the activation function.

Referring to FIG. 3, the nonlinear separation function can solve several layers of perceptons by mesh network. The MLP can perform nonlinear separation functions using a plurality of linear separation functions. The MLP can be a feed-forward neural network composed of an input layer, a hidden layer composed of hidden nodes, and an output layer. 3 is a diagram showing an MLP structure composed of an input layer composed of input values x0 - x2 and a bias bx, a hidden layer of a2 - a2, and an output layer of 01 - 02. The synapse 101 of the neuron having the structure as shown in Fig. 1 can perform n: n matching. In FIG. 3, x0 - x2 may be a neuron or an input provided by the system. a0 - a2 can be a neuron. Therefore, as shown in FIG. 3, the neurons of the input layer and the hidden layer may apply n: n matching electrical signals to the corresponding neurons, respectively.

The input layer can pass the received value directly to the hidden layer. The hidden layer may include a plurality of nodes (e.g., a0-a2 in FIG. 2B), and each node may output a signal SOP that multiplies a plurality of input signals, respectively, with weights, and then adds the product signals. The hidden layer can carry sum calculation + active function calculation and transmit it to the output layer. The output layer can perform a total acid + active function calculation to generate an output signal. That is, the principle of MLP performs a forward computation starting from the left and proceeding only to the right, and each layer (hidden layer and output layer) can perform weight sum calculation and active function calculation. The weight sum calculation may take the form of combining the nodes of the input layer or the hidden layer. The active function is a nonlinear function (sigmoid function), which can be a function that transforms the combination of input variables or hidden nodes. The complexity of the neural network algorithm can be very high. In this case, the neural network may be overfitted and the computation time may become longer. The neural network device can solve the complexity of the neural network device by using the normalization technique. The normalization method of the neural network device may be a drop-connect and / or dropout method.

4A and 4B are views for explaining a regulization method in a neural network apparatus according to various embodiments of the present invention. 4A is a diagram showing a structure of a neural network device that performs a normalization function using a drop connect method. 4B is a diagram showing a structure of a neural network device that performs a normalization function using a dropout method.

Referring to FIG. 4A, a drop connection may be a method of drop-connecting (pruning) an input signal applied to a node of a hidden layer. A drop connection can be a way to prevent some of the values from being applied when the weight is passed from one layer to the next. And may be the node output of the previous layer outputted from the node x0 (411) - x2 (415) in Fig. 4A. Node a0 (451) - a2 (452) may be the current node. 4A shows an example in which the signals 431, 433, and 435 among the output signals of the nodes x0 (411) - x (415) are drop-connected.

Referring to FIG. 4B, dropout (pruning) may mean that some of the nodes in one layer do not operate. In other words, dropout can mean the operation that does not receive the value of the previous layer and does not pass the value to the next layer. Dropout can be a way to remove randomly set concealment nodes for the training example in the MLP. 4B shows an example in which the a0 node 451 and the a2 node 455 are dropped out.

5 is a diagram illustrating a configuration of an apparatus for implementing a drop connect function in a neural network apparatus using memristors according to various embodiments of the present invention. Referring to FIG. 5, the input unit 510 may input an input signal or a computed signal. For example, the input unit 510 can receive an external input signal during the initial operation, and can receive the operation signal output from the output unit 540 during the operation period. The input unit 510 may apply an input signal to the first line.

The operation unit 530 may include a plurality of nodes that multiply input signals and corresponding weights, and add the signals. The arithmetic operation unit 530 may use a memristor (MEMRISTOR). Memristor refers to a non-linear passive, two-terminal electrical component of memory associated with the combination of charge and magnetic flux, such as resistive random access memory (ReRAM), phase change RAM (PCRAM), and magnetoresistive random access memory (MRAM) Each node of the operation unit 530 may generate multiplication signals of weights corresponding to a plurality of input signals and input signals, and then output the added signals. When a memristor is used, the product of the input signal and the weight can be represented by the current value, and the SOP signal can be represented by the sum of the current values. For example, the operation unit 520 may include a plurality of memristors that are located between the first lines and the second lines. In addition, the memristors connected to the second line respectively apply signals obtained by multiplying the weight values corresponding to the input signals to the corresponding second lines, and the products of the memristors are synthesized in the second line to form a sum of product (SOP) ). ≪ / RTI > For example, the SOP of each second memory may be the node output.

The normalization control unit 520 may drop-connect the input signal applied to the specific mem- ber of the operation unit 530. [ For example, the normalization control unit 520 may include a switch located between the first line and the input of each of the memristors, and a random signal generator. The normalization control unit 520 may control the switches selected by the random signal to release the connection of the input signal applied to the corresponding memristor.

The output unit 540 may determine whether to activate the node signals of the operation unit 530 output from the dropout control unit 520 based on the threshold value to generate an output signal.

5, the operation unit 530 may generate an SOP signal of the nodes based on the memristor. The normalization control unit 520 controls the signal input to the operation unit 530 by using a random signal, Connect function can be performed.

If SRAM based synaptic elements are used, it may be difficult to improve the degree of integration and to store analog information. ReRAM has a simple device structure and can store a lot of information, and therefore research on ReRAM-based synapses is actively conducted. ReRAM is a memory using resistors. The ReRAM may be arrays of cells that are crossed between the first and second lines, and the cell may include a resistance charging element. The resistance R value of the resistance element can be a weight value.

The operation unit 530 may have a configuration of a ReRAM cell array. When constructing the MLP, each node inputs a signal through a plurality of first lines, and generates and outputs an SOP through one second lines. That is, the configuration of the node may be a plurality of ReRAM cell arrays cross-connected to a plurality of first lines and a first line. At this time, the resistance values of the resistance elements 530 of the ReRAM cell array connected to one second lines may be set to different values (weight values), respectively.

6 is a diagram showing a configuration of a neural network device according to various embodiments of the present invention.

Referring to FIG. 6, the input unit 510 may apply the first input signal Vin input from the outside to the first line L11 - L1N. In the calculation period, the output unit 540 may output the signal Vout fed back from the output unit 540 1 line L11 - L1n as a second input signal.

The operation unit 530 may include N * N ReRAM cells R11 to Rnn that are cross-connected between the first line L11 to L1n and the second line L21 to L2n. The ReRAM cells R11 to Rnn may have a structure as shown in FIG. 4 and each may have a unique resistance R value, and the resistance R value may correspond to a weight value (W = 1 / R). Each of the cells R11 to Rn1, R12 to R1n2, ..., R1n to Rnn connected to the second lines L21 to L2n may be cells constituting each of the node 1 (ND1) to node n (NDn) . When the input signals are respectively applied to the corresponding first lines, the ReRAM cells connected to the nodes ND1 to NDn generate a current signal (i.e., a product signal of the input signal and weight) based on the set resistance value, -L2n and output to the SOP of each node ND1-NDn in the second line L21-L2n.

The normalization control unit 520 controls the switches S11 to Snn connected between the first input lines L11 to L1n and the memristors R11 to Rnn and the switches S11 to Snn to turn on / And a random signal generator 630 for generating a selection signal Sel11-Selnn. The random signal generator 635 may control the switching of the switches S11 to Snn so as to prevent a certain ratio (for example, 50%) of the input signals from being calculated.

The output unit 540 may include a conversion unit 640 and an activation unit 645. The converting unit 640 may input the SOP of the nodes output from the switch unit 530. The SOP at this time can be a current signal. The converting unit 640 may convert the current signal into a voltage. The activating unit 645 can activate or deactivate the SOP of the node output from the converting unit 640 according to the set threshold value. The output unit 540 may apply the output signal to the second input signal Vout of the input unit 510 in the set operation interval and may be output at the end of the operation.

7 is a diagram showing a configuration of a neural network device according to various embodiments of the present invention.

7, the selectors 711 and 713 may have the configuration of the input unit 510 of FIG. 5, the memristors 731 to 734 may have the configuration of the operation unit 530 of FIG. 5, The signal generating unit 725 and the switches 721 to 724 may be configured in the normalization control unit 520 of FIG. 5 and the converters 741 and 743 and the comparators 745 and 747 may be configured as the output unit 540 ). ≪ / RTI >

The operation unit (for example, the operation unit 530 in FIG. 5) shown in FIG. 7 shows an example in which one node includes two memristors. The memristors 731 to 734 each have a unique resistance value and can generate a current signal based on the input signal and the resistance value. The currents generated by the memristors 731 - 734 (that is, the product signals of the input signal and the weight) can be applied to the corresponding second line, and the current signals of the nodes in the second line are synthesized and generated as SOP . 7, the operation unit of the first node may include memristors 731 and 732 having resistance values of R1 and R2, respectively, and the operation unit of the second node may include members having resistance values of R3 and R4, respectively Listers 733 and 734, respectively.

The neural network apparatus according to various embodiments of the present invention can perform hardware modeling using a memristor element. In FIG. 7, the memristors 731 and 732 may be constituent units of the first node, and the memristors 733 and 734 may be constituent units of the second node. The selector 711 may select one of Vin1 or Vout1 as an input signal and apply it to the line L11 and the selector 713 may select one of Vin2 or Vout2 as an input signal and apply it to the line L12. Here, the lines L11 and L12 may be the first line to which the input signal is applied.

As to the configuration of the operation unit of the first node, the memristor 731 is located at the intersection of the line L11 and the line L21 and may have the value of the resistance R1. The memristor 731 can generate a current signal based on the input signal (Vin1 or Vout1) and the resistor R1, and can apply it to the line L21. The memristor 732 is located across the lines L12 and L21 and may have a value of resistance R2. The memristor 732 can generate a current signal based on the input signal (Vin2 or Vout2) and the resistor R2 and can apply it to the line L21. As to the configuration of the operation unit of the second node, the memristor 733 is located at the intersection of the line L11 and the line L22 and can have the value of the resistor R3. The memristor 733 can generate a current signal based on the input signal (Vin1 or Vout1) and the resistor R3 and can apply it to the line L22. The memristor 734 is located across the line L12 and the line L22 and may have a value of the resistor R4. The memristor 734 can generate a current signal based on the input signal (Vin2 or Vout2) and the resistor R4, and can apply the line L22. Here, the resistors R1 to R4 can correspond to the values of the weights (G = 1 / R), and the memristors can be set to their respective resistance values.

Thus, the memristors 731 and 732 can output the corresponding input signals to the line L21 by multiplying the weight values (the respective inherent resistance values). In this case, the combined current of the first node applied to the line L21 can be expressed as Equation (1).

In Equation (1), I1 may be a current value corresponding to the SOP of the first node generated on line 21, Vin1 and Vin are input signals, G1 and G2 are resistors R1 and R2 of memristors 731 and 732, Can be a weight value based on R2. In this way, I2, which is the SOP of the second node, can be generated. The neural network device according to various embodiments of the present invention can generate the SOM based on the memristor as the basic operation of the neural network algorithm.

The SOPs generated in the memristor may be represented by currents I1 and I2, and the voltage converters 741 and 743 may convert the currents i1 and i2 of the corresponding first and second nodes, respectively, to a voltage. The comparators 745 and 747, which are activated, can determine whether the voltage converted by the voltage converters 741 and 743 is activated, respectively, and output the output signals Vout1 and Vout2. The comparators 745 and 747 may perform an activation function. For example, -Vcc (supply voltage of OP amp) is output if it is smaller than a specific Vref (reference volatage), and + VCC can be output if it is larger than Vref. Where Vref can be determined by the bias voltages b and bx in Figs. 2A and 2B.

8A to 8D are views for explaining the operation of the output unit in the neural network apparatus according to various embodiments of the present invention.

8A to 8D, the activation unit may be configured using a comparator as shown in FIG. 8A. 8B shows the operational characteristics of the comparator. The comparator can have the following output characteristics.

The characteristic of the comparator as shown in FIG. 8B may have characteristics similar to those of the Sigmoid function shown in FIG. 8C. The neural network device can use a combination function that combines the input argument and an activation function that transforms the input signal by combining it. The composite function is a function for converting the input signal into one piece of information and can be weight data. At this time, the activation function is a function for transferring the composite value (for example, SOP) of the input signal to the output layer or the hidden layer, and can be a function that can convert the SOP to a certain range of values. Sigmoid function can be the most used function as active function. The sigmoid function can have characteristics that approximate a linear function when the output values are close to zero. If the comparators 745 and 747 in the neural network device have a sufficiently large dml + VCC and set -VCC to 0, it can be seen that it is similar to the ReLU activation function as shown in FIG. 8D. Depending on the operation of the comparator, various active functions can be implemented.

The output signal Vout1 output from the comparator 745 can be expressed by Equation (3) below. The output signal Vout2 can also be obtained by the following Equation (3).

The output voltages Vout1 and Vout2 output to the comparators 745 and 747 may be applied as the second input signals of the selectors 711 and 713, which are the input units. The first input signal Vin input from the outside can be selected at the time of performing the neural network device arithmetic operation and the second input signal Vout which is the calculated signal can be selected in the subsequent arithmetic section. Therefore, the operation unit can perform a calculation operation based on the first input signal Vin, and can then perform a calculation operation based on the second input signal Vout, which is an output signal. Then, the computed final output signal can be generated by performing the set number of arithmetic operations.

9 is a diagram for explaining a loop operation in which an output signal is fed back to a next layer in a neural network apparatus according to various embodiments of the present invention.

Referring to FIG. 9, the output of the first loop (1 ^st loop, 1 ^st layer) of the neural network device according to the initial input Vin can be expressed as Equation (4).

Therefore, the output Vout of the neural network apparatus can be expressed by Equation (5).

The output of the second loop (2 ^nd loop, 2 ^nd layer) of the neural network device, which receives the output signals Vout1 and Vout2 generated in the process as shown in Equation (4), is expressed by Equation Can be obtained.

As shown in Equation (4) and Equation (6), the neural network device can generate an SOP based on input signals, and generate an output signal by determining whether to activate the generated SOP. The output signal of the neural network device can be expressed as Equation (7).

Since the neural network apparatus shown in FIG. 7 has high complexity, the normalization and / or refinement functions can be added. One of the methods of normalization and / or refinement may be a drop connection. The drop connection may be a method of disconnecting the input signal applied to the memristors as shown in FIG. 4A. 7, the drop connection connects the switches S11 to Snn to the input terminals of the memristors R11 to Rnn of the operation unit 530, and generates a random signal generator (for example, a random signal generator 635 can control on / off operation of the switches S11 - Snn by the random signals Sel11-Selnn generated in the switches S11-Snn. The switch unit 630 in FIG. 6 may be a switch 721 - 724 in FIG. 7, and the random signal generator 635 in FIG. 6 may be a random signal generator 725 in FIG. The switch 721 is connected between the line L11 and the memristor 731 and can disconnect the input signal applied to the memristor 731 by the selection signal Sel1. The switch 722 is connected between the line L12 and the memristor 722 and can disconnect the input signal applied to the memristor 722 by the selection signal Sel2. The switch 723 is connected between the line L11 and the memristor 733 and can disconnect the input signal applied to the memristor 733 by the selection signal Sel3. The switch 734 is connected between the line L12 and the memristor 734 and can disconnect the input signal applied to the memristor 734 by the selection signal Sel4.

The random signal generator 735 may generate a switch control signal that can cancel the connection of some signals of the input signals. 10A and 10B are diagrams showing a configuration of a random signal generator in a neural network apparatus according to various embodiments of the present invention.

Referring to FIGS. 10A and 10B, the random signal generator 725 may use an N-bit Fibonacci linear feedback shift register (LFSR). The Fibonacci LFSR can be configured with an XOR gate that excludes a shift register and a portion of the shift register. In FIGS. 10A and 10B, the XOR gate shows a configuration example in which data positioned at the previous stage of the final output and the final output is subjected to an exclusive OR operation and applied to the input of the shift register. The random signal generator 725 may include a plurality of Fibonacci LFSRs as shown in FIG. 10A to generate switch control signals for controlling the dropout of each node. Also, the random signal generator 725 may include one Fibonacci LFSR as shown in FIG. 10B to generate switch control signals for controlling the dropout of each node. In FIGS. 10A and 10B, a dropout is applied if the switch control signal is 1, and a dropout is not applied when the switch control signal is 0. The random function generator 725 may generate a random signal such that the dropout rate (for example, the ratio of the signal output as 1) is 50%.

7, the memristor (ReRAM) 731 - 734 may have a unique resistance value (R1 - R4), and these resistance values may be changed . And the resistance values of the memristor can correspond to the weight value. When the inputs Vin1 and Vin2 are input in the first loop, the memristors 731 to 734 generate current signals based on the previously set resistance values R1 to R4, and these current signals are applied to the lines 21 and 22 Node and the second node) to generate currents I1 and I2 (i.e., SOP1 and SOP2). At this time, current I1 may be the sum of the current generated by memristor 731 by R1 and Vin1 and the current generated by memristor 732 by R2 and Vin2. The current I1 can be the output of one node of the first layer of the neural network device. The current I2 can be the sum of the current generated in the memristor 733 by R3 and Vin1 and the current generated in the memristor 732 by R4 and Vin2. Current I2 can be the output of another node of the first layer of the neural network device.

The neural network device can improve the performance of the algorithm through the drop connect (or pruning) function. The drop connector may be made to have some of the signals input to the operation unit to be zero. The drop connection of the neural network apparatus according to various embodiments of the present invention may use a method of making the signals input to a part of the memristors constituting the operation unit 530 zero (a method of disconnecting an input signal). The selection of the input signals may be determined by the random signal generator 725. The random signal generator 725 may be implemented as an N-bit Fibonacci LFSR having the configuration shown in FIG. 10A or FIG. 10B. The switch control signal generated by the random signal generator 725 may be applied to the switches 721 and 723 to drop out the SOP of the nodes L21 and L22 applied to the voltage comparators 741 and 743. [ The voltage comparators 741 and 743 can drop out the output of the node when the input is applied to zero.

In order to apply the output of the current layer (e.g., the first layer) to the input of the next layer (e.g., the second layer), the voltage converters 741 and 743 may convert the current I1 and current I2, respectively, . At this time, the voltage converters 741 and 743 can be configured using an OP amp that converts a current into a voltage. The feedback resistances Rf of the voltage converters 741 and 743 can use any appropriate value. The outputs of the voltage conversion units 741 and 743 may be applied as comparative inputs of the comparators 745 and 747. [ Vref of the comparators 745 and 747 can be set to appropriate values. The comparators 745 and 747 compare the voltage converted by the voltage converters 731 and 733 with + Vref to output Vout1 and Vout2, respectively. The comparators 745 and 747 may change the set values of the supply voltage and Vref, and may be represented by a Relu activation function and a sigmoid function depending on the setting.

The output voltages Vout1 and Vout2 output from the comparators 745 and 747 may be the input signals of the next layer and may be applied as the second input signals to the input portion (for example, the input portion 510 of FIG. 4). Then, the input unit can select the second input signal and apply it to the first line of the operation unit. The neural network device can repeat the above-described operation by the set number of times, and can output the final output signal Vout to the outside when the preset number of times is reached.

11 is a diagram showing a configuration of another embodiment for performing a normalization operation in a neural network apparatus according to various embodiments of the present invention.

Referring to FIG. 11, an input unit 1110 may input an input signal or a computed signal. For example, the input unit 1110 can receive an external input signal during the initial operation, and can receive a signal output from the output unit 1150 during the operation period. The input unit 510 may apply an input signal to the first line.

The operation unit 1130 may include a plurality of nodes that multiply input signals by corresponding weights, and then add the signals. The operation unit 1130 may use a resistance random access memory (ReRAM). Each node of the operation unit 1130 may generate product signals of weights corresponding to a plurality of input signals and input signals, and output the added signals SOP. The operation unit 1130 may include a plurality of memristors cross-connected to the first and second lines.

The first normalization control unit 1120 may drop-connect the input signal applied to the specific mem- ber of the operation unit 530. [ For example, the first normalization control unit 520 may include a switch located between the first line and the inputs of the memristors, and a first random signal generator. The first normalization controller 1120 may control the switches selected by the random signal to release the connection of the input signal applied to the corresponding memristor.

The second normalization controller 1140 may drop out some node signals among the plurality of node signals of the operation unit 1130 according to the random control signal. For example, the second normalization controller 1140 may include switches connected to the second lines and a second random signal generator. The first normalization control unit 1140 can drop out the signal of the specific second line selected by the random signal.

The output unit 11500 can generate an output signal by determining whether to activate the node signals of the operation unit 1130 based on the threshold value.

11, the operation unit 1130 may generate SOP signals of the nodes based on the memristor, and the first normalization control unit 1120 may output the signals input to the operation unit 1130 to the first random signal And the second normalization controller 1140 can drop out the node signal output from the operation unit 11130 by the second random signal.

12 is a diagram showing a configuration of a neural network device according to various embodiments of the present invention.

12, the selectors 1211 and 1213 may have the configuration of the input unit 1110 shown in FIG. 11, the memristors 1231 to 1234 may have the configuration of the calculator 1130 shown in FIG. 11, 1 random signal generator 1225 and switches 1221 to 1224 may be configured as the first normalization controller 1120 shown in Figure 11 and the switches 1241 and 1243 and the second random signal generator 1245 11, and the converters 1251 and 1253 and the comparators 1255 and 1257 may have the configuration of the output unit 1150 shown in FIG.

The operation unit (for example, the operation unit 1130 in FIG. 11) shown in FIG. 12 shows an example in which one node includes two memristors. The memristors 1231 to 1234 each have a unique resistance value and can generate a current signal based on the input signal and the resistance value. The currents generated by the memristors 1231 to 1234 (i.e., the product signals of the input signal and the weight) can be applied to the corresponding second lines L21 and L22, and the current signals of the corresponding nodes in the second lines L21 and L22 Can be synthesized and generated as SOPs. For example, in FIG. 12, the operation unit of the first node may include memristors 1231 and 1232 having resistance values of R1 and R2, respectively, and the operation unit of the second node may include members having resistance values of R3 and R4, respectively And listers 1233 and 1234. [

The selector 1211 can select one of Vin1 or Vout1 as an input signal and apply it to the line L11 and the selector 1213 can select one of Vin2 or Vout2 as an input signal and apply it to the line L12. Here, the lines L11 and L12 may be the first line to which the input signal is applied.

As to the configuration of the operation unit of the first node, the memristor 1231 is located at the intersection of the line L11 and the line L21 and can have the value of the resistance R1. The memristor 1231 can generate the current signal based on the input signal (Vin1 or Vout1) and the resistor R1 and can apply it to the line L21. The memristor 1232 is located across line L12 and line L21 and may have a value of resistance R2. The memristor 1232 can generate a current signal based on the input signal (Vin2 or Vout2) and the resistor R2 and can apply it to the line L21.

Looking at the configuration of the operation unit of the second node, the memristor 1233 is located at the intersection of the line L11 and the line L22 and can have the value of the resistor R3. The memristor 1233 can generate the current signal based on the input signal (Vin1 or Vout1) and the resistor R3 and can apply it to the line L22. The memristor 1224 is located across line L12 and line L22 and may have a value of resistance R4. The memristor 1234 can generate the current signal based on the input signal Vin2 or Vout2 and the resistor R4, and can apply the line L23. Here, the resistors R1 to R4 can correspond to the values of the weights (G = 1 / R), and the memristors can be set to their respective resistance values.

The neural network device according to various embodiments of the present invention may have normalization and / or refinement functions. Normalization and / or refinement methods may be drop-connect and drop-out. The drop connection may be a method of drop-connecting the input signal applied to the memristors as shown in FIG. 4A. The dropout may drop out the output of the nodes output from the operation unit 1120 as shown in FIG. 4B.

12, the drop connection connects the switches 1221-1224 respectively to the inputs of the memristors 1231-1234 of the operation unit 1130, and generates the random signals Sel1 - S122 generated by the first random signal generator 1225, And by turning on / off the switches 1221-1224 by Sel4. The switch 1221 is connected between the line L11 and the memristor 1231 and can disconnect the input signal applied to the memristor 1231 by the selection signal Sel1. The switch 1222 is connected between the line L12 and the memristor 1232 and can disconnect the input signal applied to the memristor 1232 by the selection signal Sel2. The switch 1223 is connected between the line L11 and the memristor 1233 and can disconnect the input signal applied to the memristor 1233 by the selection signal Sel3. The switch 1224 is connected between the line L12 and the memristor 1234 and can disconnect the input signal applied to the memristor 1234 by the selection signal Sel4.

12, the dropout connects switches 1241 and 1243 capable of switching output of a node output to a node output terminal of the operation unit 1130, and a second random signal generated by the second random signal generation unit 1245 And can be executed by an operation of turning on / off the switch. The switch 1241 may include a first transistor T1 connected between the line L21 and the voltage converter 1251 and a second transistor T2 connected between the line L21 and the ground terminal. The switch 1241 may drop out the SOP of the first node. For example, if the switch control signal generated by the random signal generator 1245 is a first logic (e.g., high logic), the second transistor T2 is turned on and the first transistor T1 is turned off, I1 signal of L21) can be dropped out. If the switch control signal generated by the random signal generator 735 is a second logic (e.g., low logic), the first transistor T1 is turned on and the second transistor T2 is turned off, so that the SOP of the first node Signal) may be applied to the input of the voltage converter 741. [

The switch 1243 may also include a first transistor T3 connected between the line L22 and the voltage converter 1253 and a second transistor T4 connected between the line L22 and the ground. The switch 1243 may drop out the SOP of the second node. The switch 1243 may be operated in the same manner as the operation of the switch 124. [

The second random signal generator 1245 may generate a switch control signal to drop out the output signal of some of the nodes. The first random signal generator 1225 and the second random signal generator 1245 may use an N-bit Fibonacci LFSR having a configuration as shown in FIGS. 10A and 10B.

12, the memristor (ReRAM) 1231 to 1234 may have a unique resistance value (R1 to R4), and the resistance values may correspond to a weight value . When inputs Vin1 and Vin2 are input in the first loop, memristors 1231 to 1234 generate current signals (input signal and weight product signals) based on previously set resistance values R1 to R4, L21 and line L22 (i.e., the first node and the second node) and may be generated as currents I1 and I2 (i.e., SOP1 and SOP2). At this time, the current I1 may be the sum of the current generated in the memristor 1221 by R1 and Vin1 and the current generated in the memristor 1232 by R2 and Vin2.

The first normalization control unit 1120 of the neural network apparatus may perform a drop connect function by making part of the signals input to the operation unit 1130 zero. The selection of the input signals may be determined by the first random signal generator 1225. The switch control signals sel1 to sel4 generated in the first random signal generator 1225 are respectively applied to the switches 1221 to 1224 so that the input signals applied to the memristors 1231 to 1234 can be set to zero, connect). When the input signal is released by the drop connection, the corresponding memristor will not perform the arithmetic operation on the input, thereby reducing the computation time.

The second normalization control unit 1140 of the neural network apparatus may perform a dropout function by making some of the SOP signals output to the operation unit 1130 zero. The selection of the node to be dropped out may be determined by the second random signal generator 1245. The switch control signals generated by the second random signal generator 1245 may be applied to the switches 1241 and 1243 to drop out the SOP of the nodes L21 and L22 applied to the voltage comparators 1251 and 1253, respectively. The voltage comparators 1251 and 1253 can drop out the output of the node when the input is applied to zero. When the output of a specific node is dropped out, the calculation operation for the output signal of the specific node is not performed in the calculation operation of the next layer, thereby reducing the calculation time.

In order to apply the output of the current layer (e.g., the first layer) to the input of the next layer (e.g., the second layer), the voltage converters 1251 and 1253 may convert the currents I1 and I2, respectively, . At this time, the voltage converters 1251 and 1253 can be configured using an OP amp that converts a current into a voltage. The outputs of the voltage conversion units 1251 and 1253 may be applied as comparative inputs of the comparators 1255 and 1257. The Vref of the comparators 1255 and 1257 can be set to an appropriate value. The comparators 1235 and 1237 may be represented by a Relu activation function and a sigmoid function depending on the setting.

The output voltages Vout1 and Vout2 output from the comparators 1255 and 1257 may be the input signals of the next layer and may be applied as the second input signals to the input unit (e.g., the input unit 1110 of Fig. 11). Then, the input unit 1110 may select the second input signal and apply the selected second input signal to the first line of the operation unit 1230. The neural network device can repeat the above-described operation by the set number of times, and can output the final output signal Vout to the outside when the preset number of times is reached.

In FIG. 12, the dropout is located at the output terminal of the operation unit 1130 and shows an example in which the output of the selected node is dropped out. At this time, a switch may be connected between a specific selector (or a specific first line) of the input unit 1110 and the operation unit 1130, and a switch may be controlled by the second random signal generation unit 1245. That is, the signal applied to the specific first line by the selector 1110 may be the output of the specific node in the previous layer. Therefore, even if the input signal of the first line is disconnected from the input signals applied to the first lines, the same effect as the dropout operation can be obtained.

The neural network apparatus having the configuration as shown in Figs. 11 and 12 can perform the normalization operation by selecting one of the drop connect and the dropout. That is, a control unit of a neural network device, not shown, may select a drop connect or dropout function when driving a neural network device. When the drop connect function is selected, the neural network apparatus can activate the first normalization control unit 1120 and deactivate the second normalization control unit 1140 to drop-connect the input signal applied to the operation unit 1130 have. When the dropout function is selected, the neural network apparatus activates the second normalization control unit 1140 and deactivates the first normalization control unit 1120 to drop out the output (SOP signal) of the node output from the operation unit 1130 have. In addition, a control unit (not shown) can select both the drop connection and the dropout function of the neural network device. In this case, the first normalization controller 1120 and the second normalization controller 1140 are all activated in the neural network device to perform the drop connect and dropout operations.

A method of normalizing a neural network device according to various embodiments of the present invention includes the steps of: generating a plurality of first and second lines, each of which has a plurality of first lines and a plurality of second lines, Applying a plurality of input signals to the first lines and a corresponding one of the first switches connected between the first line and the second memory elements by a first random signal, A drop connect step of switching off the connection of the input signals to be applied to the memory element by switching control; and a second step of synthesizing the current signals generated by the corresponding input signal and resistance value in the memristor elements in the second lines Generating an SOP (sum of product) signal of the node and an output stage for activating signals of the second line by an activation function and feeding back the signals of the second line to the input signal It may contain.

Wherein the step of generating the SOP in the normalization method of the neural network device comprises the steps of: crossing the plurality of first lines and the plurality of second lines, wherein in the memristors having inherent resistance values, And generating an SOP of the corresponding node by synthesizing the currents generated in the memristors connected to the second line.

In the normalization method of the neural network device, the drop connect step may disconnect 50% of the input signals applied to the memristors.

In the normalization method of the neural network device, the output step may include a step of converting the current into a voltage, which is connected to the second line, and determining whether the output signal of the conversion unit is activated by the set bias voltage. The step of determining whether to be active may determine whether the voltage signal is active by comparing the converted voltage signal with a set reference voltage.

The method of normalizing a neural network device according to various embodiments of the present invention is characterized in that the second switches of the corresponding ones of the second switches connected between the second lines and the output unit by the second random signal are switched And dropping out the SOP applied to the output unit.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention. Therefore, the scope of the present invention should not be limited by the described embodiments, but should be determined by the scope of the appended claims, as well as the appended claims.

Claims (20)

An input unit for applying a plurality of input signals to corresponding first lines;
Each of the memory elements having a unique weight value, and each of the memory elements having a weight value corresponding to the input signal of the corresponding first line and a plurality of second An operation unit for generating product signals and outputting the product signals to corresponding second lines of the plurality of second lines;
A drop connect controller which switches between the first line and the memory devices and disconnects an input signal applied to the corresponding memory device by switching the switch by a random signal so as to disconnect the input signal; And
And an output unit connected to the second line for activating signals of the second line and applying the signals as an input signal to the input unit and outputting the result when the set number of operations is completed.
The method according to claim 1,
The operation unit
And a plurality of memristors crossing the switch and the second line,
Each of the memristors includes a resistance value corresponding to a weight value and generates a current corresponding to a product of the input signal and the weight based on the resistance value and outputs the current to the second line, a neural network device is generated as a sum of products.
3. The method of claim 2,
The drop connect control unit
A random signal generator for generating a random signal; And
And switches connected between the first line and the memristor elements, respectively,
And a corresponding switch is switched and controlled based on the random signal to release a connection of an input signal applied to the memristors.
The method of claim 3,
The random signal generator
A neural network device that is a Fibonacci linear feedback shift register.
5. The method of claim 4,
The drop connect control unit
And disconnects 50% of the input signals applied to the memory devices.
The method of claim 3,
The output
A converter connected to the second line to convert a current into a voltage; And
And an activation unit for performing an activation operation of an output signal of the conversion unit by a set bias voltage.
The method according to claim 6,
Wherein the activation unit is a comparator.
An input unit for applying a plurality of input signals to corresponding first lines;
Each of the memory elements having a unique weight value, and each of the memory elements having a weight value corresponding to the input signal of the corresponding first line and a plurality of second An operation unit for generating product signals and outputting the product signals to corresponding second lines of the plurality of second lines;
A drop connect control unit which switches between the first line and the memory devices and disconnects an input signal applied to the corresponding memory device by switching the switch by a first random signal;
A dropout control unit including switches connected to the second lines, the dropout control unit dropping out at least one signal of the second line by a second random signal; And
And an output unit connected to the second line for activating signals of the second line and applying the signals as an input signal to the input unit and outputting the result when the set number of operations is completed.
9. The method of claim 8,
The operation unit
And a plurality of memristors cross-connected to the plurality of first lines and the plurality of second lines,
Each of the memristors includes a resistance value corresponding to a weight value and generates a current corresponding to a product of the input signal and the weight based on the resistance value and outputs the current to the second line, a neural network device is generated as a sum of products.
10. The method of claim 9,
The drop connect control unit
A first random signal generator for generating a first random signal; And
And first switches connected between the first line and the memristor elements, respectively,
And a first switch corresponding to the first random signal is switched and controlled to release a connection of an input signal applied to the memristors.
11. The method of claim 10,
The dropout control unit
A second random signal generator for generating a second random signal; And
And second switches respectively connected between the second lines and the output unit,
And the second switch corresponding to the second random signal is switching controlled to release the SOP of the second line applied to the output unit.
11. The method of claim 10,
Wherein the first random signal generator and the second random signal generator
A neural network device that is a Fibonacci linear feedback shift register.
13. The method of claim 12,
The output
A converter connected to the second line to convert a current into a voltage; And
And an activation unit for performing an activation operation of an output signal of the conversion unit by a set bias voltage.
Applying a plurality of input signals to the first lines of memory elements, each of the memory elements being located at a plurality of first lines and a plurality of second lines, each having an intrinsic resistance value corresponding to a respective weight value;
And a first part of the first switches connected between the first lines and the memory elements is switched by a first random signal to switch off the connection of the input signals applied to the memory elements Drop connect phase;
Synthesizing current signals generated by the input signal and the resistance value corresponding to the memory elements in the second lines to generate a signal of the node; And
And activating the signals of the second lines by an activation function to feed back the signals of the second lines to the input signal.
15. The method of claim 14,
The step of generating the signal of the node
Generating a current corresponding to a product of an input signal and a weight based on a resistance value of memristors crossing the plurality of first lines and the plurality of second lines and having resistances inherent thereto; And
And generating a sum of product (SOP) signal of a corresponding node by synthesizing currents generated in memristors connected to the second lines.
16. The method of claim 15,
The drop connect step
And removing 50% of the input signals from the input signals applied to the memristors.
17. The method of claim 16,
The output step
Converting the current into a voltage coupled to the second lines; And
And determining whether the output signal is active by the set bias voltage.
18. The method of claim 17,
The step of determining whether to activate
And comparing the converted voltage signal with a set reference voltage to determine whether the voltage signal is active.
16. The method of claim 15,
Further comprising the step of dropping out the SOP applied to the output unit by controlling switching of a corresponding part of the second switches among the second switches connected between the second lines and the output unit by the second random signal Normalization method.
20. The method of claim 19,
The output step
Converting the current into a voltage coupled to the second lines; And
And determining whether the output signal is active by the set bias voltage.

Images