KR20210051288A - Neural network learning device comprising analog batch normalization device - Google Patents

Abstract

Disclosed is a neural network learning device capable of reducing a load applied to hardware by reducing a computation amount of a processor. According to various embodiments, the neural network learning device includes: an input unit to input training data for training an artificial neural network model; and a learning processor to train the artificial neural network model using the training data. The artificial neural network model includes a plurality of layers to receive a signal corresponding to the data received from the input unit and at least one batch normalization device interposed between the plurality of layers. The batch normalization device includes a resistance memory array (RRAM array) to perform a convolution operation for a first analog signal received from a first layer of the plurality of layers and a plurality of active device to transmit a third analog signal to a second layer by normalizing a second analog signal subject to the convolution operation based on the resistance memory array.

Description

Neural network learning device including an analog batch normalization device {NEURAL NETWORK LEARNING DEVICE COMPRISING ANALOG BATCH NORMALIZATION DEVICE}

Various embodiments relate to a neural network training apparatus including an analog batch normalization apparatus. Specifically, the neural network learning apparatus may include a resistive memory arrangement for batch normalization and a circuit arrangement having an active element for normalization.

As technology related to artificial intelligence develops, artificial intelligence technology is being applied in various industries. Recently, as the performance of processing devices such as a processor and a dedicated accelerator can be improved, a neural network learning device or a neural network learning algorithm requiring a large amount of computation can be driven, so that an electronic device to which artificial intelligence is applied can operate a limited function in the related art.

The object recognition algorithm using deep learning employs a convolution neral network (CNN) or a deep neral network (DNN) model composed of a convolution layer.

In the case of a model using a convolutional layer or a convolutional layer, there is a problem that is greatly affected by a change in distribution of input values of each layer.

For example, according to a change in the distribution of the input values, a difference in the distribution of the output values of the next layer is caused, and an error amount of the result value is increased. The artificial neural network learning apparatus has a fear of loss or explosion in the artificial neural network training process due to the change in distribution. To solve this problem, the artificial neural network learning apparatus introduces a normalization algorithm.

Since the batch normalization method is trained through normalization in batch units, it is possible to reduce the movement of internal covariates, and the result values are also rapidly converged and widely used. The covariate shift problem due to each internal covariate can be solved by a method of normalizing the output values of each layer.

After the convolutional neural network is trained by applying batch normalization from the learning stage, batch normalization can be applied to each output value or input value of each layer of the convolutional neural network hardware.

Since the batch normalization method uses statistical values of data in batch units, a storage space for storing many operations and calculation values is required. The batch normalization method implements batch normalization through approximation with extensive operations. Excessive approximation can lead to performance degradation of the entire neural network learning system.

Since the digital arrangement normalization method requires a digital signal processor (DSP) between layers, additional hardware is required, and a signal path may be complicated accordingly.

Therefore, even in the use of the batch normalization method, a neural network learning apparatus is required to reduce excessive approximation and reduce hardware such as a digital signal processor.

A neural network training apparatus according to various embodiments includes an input unit for inputting training data for training an artificial neural network model, and a learning processor for training the artificial neural network model using the training data, and the artificial neural network model comprises: A plurality of layers receiving a signal corresponding to the data received from the input unit, and at least one batch normalization device disposed between the plurality of layers, wherein the batch normalization device comprises the plurality of layers. A third analog signal to a second layer by normalizing a resistive memory array (RRAM array) that performs convolutional processing of the first analog signal transmitted from the first layer among the layers, and a second analog signal that is convolutional-processed from the resistive memory array. It may include a circuit arrangement including a plurality of active elements for transmitting.

An electronic device according to various embodiments of the present disclosure includes a batch normalizing device for normalizing an artificial neural network model, and the batch normalizing device is a resistive memory array for convolutional processing a first analog signal transmitted from a first layer among the plurality of layers. (RRAM array), a circuit arrangement including a plurality of active elements for transmitting a third analog signal to a second layer by normalizing a second analog signal subjected to convolution from the resistive memory array, and a control unit configured to control the circuit arrangement It may include.

The neural network learning apparatus according to various embodiments can output an analog signal by implementing batch normalization as an active device, thereby reducing the use of a digital signal processing device.

The apparatus for learning a neural network according to various embodiments may reduce the amount of computation of a processor, thereby reducing a load applied to hardware.

The apparatus for learning a neural network according to various embodiments may implement batch normalization without deteriorating network performance.

The neural network training apparatus according to various embodiments may reduce a covariate movement problem caused by an internal covariate of a resistance memory.

1 is a block diagram of an apparatus for learning an artificial neural network, according to various embodiments.
2 is a conceptual diagram of an artificial neural network model constituting a neural network training apparatus according to various embodiments.
3A and 3B illustrate a normalization operation to reduce an error that occurs.
4 is a block diagram of a kernel matrix included in an artificial neural network model according to various embodiments.
5 is a conceptual diagram of a resistive memory arrangement implementing a kernel matrix included in an artificial neural network model, according to various embodiments.
6A and 6B are circuit diagrams of a batch normalization apparatus, according to various embodiments.
7A to 7E illustrate switching operations in each step of an apparatus for learning a neural network according to various embodiments.
8 is a graph comparing deviations of result values through a neural network learning apparatus according to various embodiments.

1 is a block diagram of an apparatus 100 for learning a neural network according to various embodiments.

The artificial neural network may refer to hardware, software, or a combination thereof for providing a generalized output for an input.

For example, an artificial neural network is an application for simulating a convolutional neural network (CNN), a Markov chain, or a binarized neural network (BNN), and a processor for executing the application. Can work based on

Referring to FIG. 1, the apparatus 100 for learning a neural network is a device capable of performing machine learning through training, and may include a device for learning using a model composed of an artificial neural network. For example, the neural network device 100 may be configured to input, output, build a database, and store information used for data mining, data analysis, and machine learning algorithms (eg, deep learning algorithms). I can.

The neural network device 100 may transmit and receive data with an external electronic device (not shown) through a communication unit (not shown), and may derive a result value by analyzing or learning data transmitted from the external electronic device. The neural network device 100 may distribute and process an operation of an external electronic device.

The neural network device 100 may be implemented as a server. In addition, the neural network device 100 may be configured in plural to form a neural network device set. Each neural network device 100 may distribute and process an operation, and may derive a result value through data analysis and learning based on the distributed data. The neural network device 100 may transmit a result obtained by using a machine learning algorithm or the like to an external electronic device or another neural network device.

According to various embodiments, the neural network device 100 may include an input unit 110, a processor 120, a memory 130, and a learning processor 140.

According to various embodiments, the input unit 110 may obtain input data for deriving an output value through training an artificial neural network model. The input unit 110 may obtain unprocessed input data. The processor 120 or the learning processor 140 may generate training data that can be input to training an artificial neural network model by pre-processing the raw input data. The preprocessing may be to extract feature points from input data. As described above, the input unit 110 may receive data through a communication unit (not shown) to obtain input data or pre-process the data.

According to various embodiments, the processor 120 may collect usage history information from the neural network learning apparatus 100 and store it in the memory 130. The processor 120 may determine the best combination for executing a specific function through the stored usage history information and predictive modeling. The processor 120 may receive image information, audio information, data, or user input information from the input unit 110.

According to various embodiments, the processor 120 may collect information in real time, process or classify the information, and store the processed information in the memory 130, the database of the memory 130, or the learning processor 140. .

According to various embodiments, when the operation of the neural network learning apparatus 100 is determined based on data analysis and machine learning algorithms, the processor 120 controls components of the neural network learning apparatus 100 to execute the determined operation. I can. Further, the processor 120 may perform the determined operation by controlling the neural network learning apparatus 100 according to a control command.

When a specific operation is performed, the processor 120 analyzes historical information indicating execution of a specific operation through data analysis and machine learning algorithms and techniques, and performs an update of previously learned information based on the analyzed information. I can. The processor 120, together with the learning processor 140, may improve accuracy of data analysis and machine learning algorithms and performance based on updated information.

According to various embodiments, the memory 130 may store input data obtained by the input unit 110, learned data, or a learning history. The memory 130 may store the artificial neural network model 131.

According to various embodiments, the artificial neural network model 131 may be stored in a space allocated to the memory 130. The space allocated to the memory 130 stores the artificial neural network model 131 being trained or learned through the learning processor 140, and when the artificial neural network model 131 is updated through learning, the updated artificial neural network model (131) can be saved. The space allocated to the memory 130 may be divided and stored into a plurality of versions according to a learning time point or a learning progress, etc. of the learned model.

According to various embodiments, the memory 130 may include a database capable of storing and classifying input data and learned data acquired by the input unit 110.

According to various embodiments, the learning processor 140 learns the artificial neural network model 131 by directly acquiring the data obtained by preprocessing the input data obtained by the processor 120 through the input unit 110 or The artificial neural network model 131 may be trained by acquiring preprocessed input data stored in the database. For example, the learning processor 140 may repeatedly learn the artificial neural network model 131 using various learning techniques to obtain the optimized parameters of the neural network model 131.

According to various embodiments, the learned model may update the artificial neural network model 131 in a database. The learning processor 140 may be integrated into the neural network learning apparatus 100 or may be implemented in the memory 130. Specifically, the learning processor 140 may be implemented using the memory 130.

According to various embodiments, the learning processor 140 generally identifies, indexes, categorizes, manipulates, stores, retrieves and outputs data for use in supervised or unsupervised learning, data mining, predictive analytics, or other devices. It may be configured to store hazardous data in one or more databases. Here, the database may be implemented using the memory 130, a memory maintained in a cloud computing environment, or another remote memory location accessible by the terminal through a communication method such as a network. It may be utilized by the processor 120 using any of a variety of different types of data analysis algorithms and machine learning algorithms. For example, examples of such algorithms include k-recent adjacency systems, fuzzy logic (e.g. probability theory), neural networks, Boltzmann machines, vector quantization, pulsed neural networks, support vector machines, maximum margin classifiers, hill climbing, derivation. Logical system Bayesian network, Peritnet (e.g. finite state machine, milli machine, Moore finite state machine), classifier tree (e.g. perceptron tree, support vector tree, Markov tree, decision tree forest, random forest), readout model And systems, artificial fusion, sensor fusion, image fusion, reinforcement learning, augmented reality, pattern recognition, automated planning, etc.

An external electronic device or an electronic device according to various embodiments disclosed in this document may be various types of devices. The electronic device may include, for example, a portable communication device (eg, a smartphone), a computer device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

Various embodiments of the present document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutes of the corresponding embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items unless clearly indicated otherwise in a related context. In this document, “A or B”, “at least one of A and B”, “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “A Each of phrases such as "at least one of B, or C" may include all possible combinations of items listed together in the corresponding phrase among the phrases. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the component from other Order) is not limited. Some (eg, a first) component is referred to as “coupled” or “connected” to another (eg, a second) component, with or without the terms “functionally” or “communicatively”. When mentioned, it means that any of the above components may be connected to the other components directly (eg by wire), wirelessly, or via a third component.

The term "module" used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits. The module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) that can be read by a machine (eg, electronic device 101). It may be implemented as software (for example, the program 140) including them. For example, the processor (eg, the processor 120) of the device (eg, the electronic device 101) may call and execute at least one command among one or more commands stored from a storage medium. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term refers to the case where data is semi-permanently stored in the storage medium. It does not distinguish between temporary storage cases.

According to an embodiment, a method according to various embodiments disclosed in the present document may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store ^TM ) or two user devices It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones), online. In the case of online distribution, at least some of the computer program products may be temporarily stored or temporarily generated in a storage medium that can be read by a device such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular number or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallel, repeatedly, or heuristically executed, or one or more of the operations may be executed in a different order or omitted. Or one or more other actions may be added.

2 is a conceptual diagram of an artificial neural network model constituting a neural network training apparatus according to various embodiments.

Referring to FIG. 2, in a general convolutional neural network, feature extraction 11 of input data 210 using a convolutional layer 201 and a pooling layer 202 and input using a fully connected layer 290 It can be used for classification 12 of data 210.

According to various embodiments, the convolutional layer 101 may be a layer for extracting meaningful features of the input data 210 through a convolution operation. For example, the convolutional layer 201 may generate new data to be transmitted to the next layer by applying a filter of a specific size or a kernel (weight) matrix 230 to the input data 210. The input/output data of the convolutional layer 201 may be referred to as feature maps.

When data input to the convolutional neural network model is an input image including a plurality of components such as an RGB component, the input data may be composed of a plurality of channels. For example, input/output data of the convolutional layer 201 may include channels other than a space of a 2D image, and a feature map of the input/output data may be formed in a 3D form.

According to various embodiments, the pooling layer 202 may reduce data received through sub-sampling. For example, the pooling layer 202 may reduce the size of data by sampling data through a pooling technique such as max pooling and average pooling.

According to various embodiments, the fully connected layer 290 is a layer for performing data classification based on features transmitted through the convolutional layer 201 and the pooling layer 202, and a feature map in a three-dimensional form is flattened. One-dimensional data can be input. As described above, data in a one-dimensional form that has passed through the fully connected layer 290 may be converted into an output signal through an activation function. Convolutional neural networks can use convolutional layers and pooling layers to maintain the three-dimensional shape of feature maps for input data (e.g., input images), preventing loss of information about the relationship between pixels or channels in the input image. Thus, the image recognition rate can be increased.

3A and 3B illustrate a normalization operation to reduce an error that occurs.

In terms of convolutional neural network models, internal covariates can cause unexpected variance of output values in each layer. In machine learning operations, the output value of each layer may show a difference from the offline output value. Since the corariate shift of the output value of each layer is used as the input value of the next layer, the amount of variance of the output value may cause an error of the final result value.

The convolutional neural network can solve the problem of covariate shift due to internal covariates by batch normalizing the result values derived from each layer. After training a neural network training model by applying batch normalization from the learning stage, batch normalization can be applied to each output value of each layer of the synthetic neural network model.

Referring to FIG. 3A, in the offline learning step, the input layer 310 may receive an input vector matrix and derive a first result value. The first convolutional layer 320 and the second convolutional layer 330 may generate output values without covariate movement in the initial model. The second convolutional layer 330 may transmit a value to the fully connected layer 340 based on the output value. Dispersion of output values obtained from the fully connected layer may occur uniformly.

According to various embodiments, when an inference operation is performed through weight extraction, covariate movement may occur in each layer. For example, an input vector matrix that has passed through the input layer 310 may have an output value shifted through a convolution process, resulting in undesired variance. The output value of the input layer 310 is used as an input value of the first convolutional layer 320, and a variance of the output value accordingly may be added. While passing through each layer, the movement of the output values becomes larger, and the error of the final result value may further increase.

Referring to FIG. 3B, a convolutional neural network including a batch normalization process shows an operation of reducing an error while undergoing a normalization process.

Before normalization, the average of the values input to the input layer 310 is 0 (μ=0) and the deviation is 1 (σ=1). The average of the output value of the input layer 310 processed in the input layer 310 or the input value of the first convolutional layer 320 is μ'(μ= μ'), and the deviation is σ'(σ=σ') It is obtained in the state of being.

The average of values input to the first convolutional layer 320 is μ'(μ = μ'), and the deviation is input in a state of σ'(σ = σ'). The average of the output value of the second convolutional layer 330 processed on the first convolutional layer 320 or the input value of the second convolutional layer 330 is μ"(μ=μ"), and the deviation is σ"(σ =σ"). Accordingly, an error occurs in the fully connected layer 340 obtained based on the result value transmitted from the second convolutional layer 330.

When each layer undergoes a normalization process, the average of the values input to the input layer 310 is 0 (μ=0) and the deviation is 1 (σ=1). After a normalization operation, the average of the output value of the input layer 310 processed to the input layer 310 or the input value of the first convolutional layer 320 is 0 (μ=0), and the deviation is 1 (σ=1). ).

The average of the values input to the first convolutional layer 320 is 0 (μ=0), and the deviation is 1 (σ=1). After the normalization operation, the average of the output value of the second convolutional layer 330 processed in the first convolutional layer 320 or the input value of the second convolutional layer 330 is 0 (μ=0), and the deviation is 1 It is obtained in the state of (σ=1). Accordingly, the value of the fully connected layer 340 obtained based on the result value transmitted from the second convolutional layer 330 may be obtained as a uniform dispersion value without error.

4 is a block diagram of a kernel matrix included in an artificial neural network model according to various embodiments.

Referring to FIG. 4, the arrangement normalization apparatus 400 may include a kernel matrix 430, a normalization circuit arrangement 410, and a control unit 420. The kernel matrix 430 may extract a feature value by assigning a weight to the input data. For example, features may be extracted from the input vector 431 transmitted from the first layer 480 through a convolution operation. The kernel matrix 430 may generate first output data 411 delivered to the second layer 490. Output data obtained through the kernel matrix 430 may be referred to as a feature map.

According to various embodiments, the kernel matrix 430 may be implemented through an RRAM crossbar array (or RRAM synapse array) of resistive memories. Resistive memory (ReRAM) can implement synaptic learning by using a characteristic in which a resistance value changes according to a voltage pulse applied from an external device. The kernel matrix 430 formed by the cross arrangement of resistive memories may obtain an output value output by an applied voltage pulse.

According to various embodiments, the first output data 411 may undergo an internal covariate shift during a convolution operation for weighting. The normalization circuit arrangement 410 normalizes the first output data 411 and normalizes the second output data in order to reduce weight loss or weight explosion caused by a change in distribution due to a parameter change of the first output data 411 Acquires 412 and transfers it to the second layer 490.

According to various embodiments, the normalization circuit arrangement 410 may include a circuit arrangement formed of an active element OP-AMP, a capacitor, and a plurality of switches. The circuit arrangement 410 may obtain an average of an output value by a switching operation, obtain a deviation between an input value and an average value, a multiple of the deviation, and the like, and implement normalization with a combination of the obtained values.

According to various embodiments, by receiving an analog signal (eg, voltage), a kernel matrix formed of a resistive memory array may be given a weight and normalized through a circuit array. For example, a kernel receiving an analog signal may process data and transmit an analog output signal without converting a digital time signal.

According to various embodiments, the controller 420 may be configured to transmit a signal for controlling the switching operation of the normalization circuit arrangement 410. The controller 420 may be electrically connected to the processor 120.

According to various embodiments, the processor 120 may transmit a control signal 421 including a clock signal and a weight signal to the controller 420. The controller 420 may transmit a plurality of clock signals 422 for controlling a switch operation based on the control signal 421. Switching is operated according to the clock signal, and the normalization circuit arrangement 410 may obtain an average value, a deviation, and a weighted value.

According to various embodiments, the normalization circuit arrangement 410 may obtain the second output data 412 by normalizing the first output data 411 according to the transmitted clock signal 422. The second output data 412 may be used as input data of the second layer 490.

5 is a conceptual diagram of a resistive memory arrangement implementing a kernel matrix included in an artificial neural network model, according to various embodiments.

Referring to FIG. 5, the kernel matrix 430 (or the resistive memory array) may arrange resistive memories in a lattice form.

According to various embodiments, the resistive memory 531 may implement synaptic learning by using a characteristic in which a resistance value changes according to an externally applied voltage pulse. The kernel matrix 430 formed by the cross arrangement of resistive memories may obtain an output value output by an applied voltage pulse. The output value may be feature points obtained from the input layer. For example, the output value is a convolutional layer converted from input data (eg, input data 210 of FIG. 2) through a weighted kernel matrix 430 (eg, convolutional layer 201 of FIG. 2). It can be an input value of.

According to various embodiments, if the number of arrays that can be integrated is limited, the plurality of resistive memory arrays may extend the required kernel of the convolutional neural network through interconnection.

6A and 6B are circuit diagrams of a batch normalization apparatus, according to various embodiments.

6A and 6B, a batch normalization device 600 includes a kernel matrix 430 formed by a cross-array of resistive memories 531 and an integrator 611 connected to one end of the kernel matrix 430. ), a capacitor 620 connected to the output terminal of the integrator 611, and a switch arrangement 630 connected in parallel with the capacitor 620 at the output terminal of the integrator 611.

The normalization operation can be operated as follows. The voltage at the output terminal of the integrator passes through the array 610 of the integrator 611 and may have values of V1, V2, V3, and V4 at each output terminal (S601). The average of the voltages at the output terminals of each integrator 611 may be connected to each other in parallel through the switching operation of the switch arrangement 630. A voltage Vμ may be applied to one end of the capacitors 620 having values CL1, CL2, CL3, and CL4 (S602). The values of CL1, CL2, CL3, and CL4 of each capacitor may be the same. The average value Vμ of the voltage can be obtained through the following equation.

,

,

Using the capacitors included in the switch array 630 and the OP AMP, Vμ and Vi may be charged in the capacitor (S603). Vi' can be applied to the output terminal by a capacitor connected in parallel to the output terminal and the input terminal of the OP AMP, and these values can be used as an output value (S604). The weight α of the Vi-Vμ value can be determined by adjusting the values of the capacitors. In operation S604, the voltage Vi' applied to the output terminal of the OP AMP may be obtained through the following equation.

,

,

The arrangement normalization apparatus of FIG. 6B shows a combination of a switch arrangement 630 circuit and an integrator 610 circuit in order to reduce the number of OP AMPs that consume a lot of power.

According to various embodiments, the combined switch arrangement 630' circuit can reduce the OP AMP included in the circuit of the switch arrangement 630. A detailed operation of the switch arrangement will be described with reference to FIGS. 7A to 7E to be described later.

7A to 7E illustrate switching operations in each step of an apparatus for learning a neural network according to various embodiments.

The switch circuit arrangement 630 may control a switching operation by receiving a control signal from the controller. The switch circuit arrangement includes the integrators 750a and 750b, and the capacitor 720a, and an electrical path between the integrators 750a and 750b and the capacitors 720a and 720b may be changed according to a switching operation.

Referring to FIG. 7A, when the switch circuit arrangement 630 receives a first signal for integration from the controller, the capacitors 720a and 720b are electrically connected to the output terminals of the integrators 750a and 750b, and a plurality of The switching circuits 701a and 701b of may be shorted to each other. Specifically, the switches 721a and 721b connected to the output terminals of the integrators 750a and 750b included in the switch circuit arrangement 630 may be short-circuited. Each of the switching circuits 701a and 701b included in the switch circuit arrangement 630 may be maintained in an open state by the opening of the switch 722. The switch circuit arrangement 630 charges the current generated from the resistive memory array (eg, the kernel matrix 430 of FIG. 4) to the capacitor 715a and the additional capacitor 720a included in the integrators 750a and 750b. I can. The capacitors 715a and the additional capacitors 720a included in the integrators 750a and 750b may have the same size. Accordingly, the amount of charge charged in the capacitor 715a may be the same.

Referring to FIG. 7B, when the switch circuit arrangement 630 receives a second signal for obtaining an average value from the controller, one end of the capacitors 720a and 720b is electrically connected to the input terminals of the integrators 750a and 750b, and a plurality of The switching circuits 701a and 701b of may be connected in parallel. Specifically, the capacitors 720a and 720b of each channel included in the switch circuit arrangement 630 are open at the output terminals of the integrators 750a and 750b, and each of the capacitors 720a and 720b may be short-circuited to each other. For example, the switches 721a and 721b connected to the output terminals of the integrators 750a and 750b are opened, and the capacitors 720a and 720b of each of the switching circuits 701a and 701b are connected to each other by a short circuit of the switch 722a. I can.

According to various embodiments, each of the capacitors 720a and 720b may be charged with V1, V2, and V3?? in FIG. 7A. After the switching operation of FIG. 7B, since each of the capacitors 720a and 720b has the same capacitance, the capacitors 720a and 720b are charged with the average value of voltage Vμ, and the output terminals of each integrator 750a and 750b are V1 and V2 as in FIG. 7A. , Can be maintained as V3??

Referring to FIG. 7C, when receiving a third signal for shifting by an average value from the control unit, the switch circuit arrangement 630 electrically connects one end of the capacitors 720a and 720b to the input terminals of the integrators 750a and 750b. The other ends of the capacitors 720a and 720b may be grounded.

According to various embodiments, after the operation of FIG. 7B, the capacitors 720a and 720b of each channel are stored as much as the average value Vμ of the output terminal. At this time, the capacitors 720a and 720b stored as much as the average value Vμ may be connected to the input terminals of the integrators 750a and 750b by shorting the switches 723a and 723b. In addition, the current flowing into the input terminals of the integrators 750a and 750b partially flows into the capacitors 720a and 720b, and the output value of the integrators 750a and 750b is shifted by the average value Vμ, and thus V1-Vμ, V2 -May be Vμ.

Referring to FIG. 7D, when the switch circuit arrangement 630 receives a fourth signal for sampling from the control unit, the switch connects the other end of the capacitor to the input terminal of the integrator when the signal transmitted from the control unit is a fourth signal. However, one end of the capacitor is grounded, and a plurality of circuits may be open to each other.

According to various embodiments, the capacitors 720a and 720b of each channel are electrically connected to the output terminals of the integrators 750a and 750b, and the plurality of switching circuits 701a and 701b may be shorted to each other. Specifically, the output terminals of the integrators 750a and 750b included in the switch circuit arrangement 630 and the switches 724a and 724b connected to one end of the capacitors 720a and 720b may be short-circuited. The switches 725a and 725b connected to the other ends of the capacitors 720a and 720b may be short-circuited to be grounded. Each of the switching circuits 701a and 701b included in the switch circuit arrangement 630 may be maintained in an open state by the opening of the switch 722. The sizes of the capacitors 715a and 715b and the additional capacitors 720a and 720b included in the integrators 750a and 750b may be the same. Accordingly, the amount of charge charged in the capacitors 720a and 720b may be CL(V1-Vμ) and CL(V2-Vμ) values.

When the switch circuit arrangement 630 receives the fifth signal for amplification from the controller, after the switching operation by the fourth signal, one end of the capacitors 720a and 720b is electrically connected to the input terminal of the integrator, and the capacitor 720a, 720b) can be grounded.

According to various embodiments, after the switching operation by the fourth signal, the capacitors 720a and 720b of each channel may be stored as much as V1-Vμ and V2-Vμ. The switches 724a, 724b, 725a, and 725b that were connected by the switching operation by the fourth signal are opened, and the capacitors 720a and 720b stored as much as the average values V1-Vμ and V2-Vμ at the same time are the switches 723a and 723b. With a short circuit of, it may be connected to the input terminals of the integrators 750a and 750b. In this case, differently from the switching operation by the fourth signal, the terminal portions connected to the ground portions of the capacitors 720a and 720b may be connected in reverse. In addition, the current flowing into the input terminals of the integrators 750a and 750b partially flows into the capacitors 720a and 720b, and the output value from the integrators 750a and 750b is amplified by the average value V1-Vμ, and 2 (V1- Vμ), 2 (V2-Vμ) may be.

According to various embodiments, the fourth and fifth signal operations may be repeatedly performed, and output values from the integrators 750a and 750b when repeated once are 2 (V1-Vμ) and 2 (V2- Vμ), the output value from the integrator (750a, 750b) when repeated twice is 4(V1-Vμ), 4(V2-Vμ), the output value from the integrator (750a, 750b) when repeated 3 times 8 (V1-Vμ), 8 (V2-Vμ) can be obtained.

According to various embodiments, the number of cycles may be determined by a control signal (eg, the control signal 421 of FIG. 4) applied from the controller 120. The number of cycles may be an amplification factor in the normalization assumption.

Referring to FIG. 7E, when the switch circuit arrangement 630 receives a sixth signal for reset from the controller, the capacitor and the output terminal of the integrator may be grounded.

According to various embodiments, the switches 726a and 726b connected to both ends of the capacitors 720a and 720b may be short-circuited. Both ends of the capacitors 720a and 720b are grounded to discharge all stored charges.

Switches 727a and 727b connected to one end of the integrators 750a and 750b may be short-circuited. The other ends of the switches 727a and 727b are grounded, so that the capacitors 715a and 715b disposed in the integrators 750a and 750b may also be discharged. Through this, the signal input after the normalization operation can be initialized.

The neural network training apparatus according to the various embodiments described above may perform batch normalization only with an analog signal without a digital processing device. According to various embodiments of the present disclosure, the arrangement normalization device may be formed of an OP AMP, a plurality of capacitors, and a plurality of switches, and may be designed to grant an average value, a deviation, and a weighting by switching operation.

8 is a graph comparing deviations of result values through a neural network learning apparatus according to various embodiments.

The graph in the first row is a graph showing the distribution of the simulated output values of the first layer and the second layer, assuming that there is no covariate. The graph in the second row is a graph showing the distribution of simulated output values of the first layer and the second layer when there is a device-dependent covariate. Referring to the two graphs, when there is a covariate due to the device, it can be seen that the deviation from the first graph remains severe.

The graph in the third row is a graph showing the distribution of output values of the first layer and the second layer obtained through normalization. Compared to the graph arranged in the first row, there is almost no deviation, and it can be seen that the frequency of each output value is similar. Through the above simulation, it can be seen that the neural network learning apparatus according to various embodiments may have a result with high accuracy.

The neural network training apparatus according to the various embodiments described above includes an input unit for inputting training data for training an artificial neural network model, and a learning processor for training the artificial neural network model using the training data, and the artificial neural network model Includes a plurality of layers receiving a signal corresponding to the data received from the input unit, and at least one batch normalization device disposed between the plurality of layers, wherein the batch normalization device comprises: A resistive memory array (RRAM array) that performs convolutional processing of a first analog signal transmitted from a first layer among the plurality of layers, and a second analog signal convolutional-processed from the resistive memory array is normalized to form a second layer. 3 A circuit arrangement including a plurality of electrical elements and a plurality of switches for transmitting analog signals may be included.

According to various embodiments, the circuit arrangement may control a voltage value transmitted through each channel of the resistive memory arrangement.

According to various embodiments, the arrangement normalization apparatus may include a control unit that controls the circuit arrangement to normalize signals applied to each channel of the resistive memory arrangement.

According to various embodiments, the control unit may transmit a control signal for a switch operation of the circuit arrangement to the circuit arrangement.

According to various embodiments, the plurality of electrical elements of the circuit arrangement include a plurality of integrators and a plurality of capacitors, and the switch may change an electrical path between the integrator and the capacitor based on the control signal of the controller. have.

According to various embodiments, in the circuit arrangement, a plurality of circuits corresponding to a channel receiving an output second analog signal or a third analog signal may be connected in parallel, and the capacitor may be connected to an input terminal or an output terminal of the integrator.

According to various embodiments, when the signal transmitted from the control unit is a first signal, the switch may electrically connect the capacitor to the output terminal of the integrator, and a plurality of circuits may be shorted to each other.

According to various embodiments, when the signal transmitted from the controller is the second signal, the switch may electrically connect one end of the capacitor to the input terminal of the integrator, and a plurality of circuits may be connected in parallel.

According to various embodiments, when the signal transmitted from the controller is a third signal, the switch may electrically connect one end of the capacitor to the input terminal of the integrator and ground the other end of the capacitor.

According to various embodiments, when the signal transmitted from the controller is a fourth signal, the switch electrically connects the other end of the capacitor to the input terminal of the integrator, grounds one end of the capacitor, and the plurality of circuits are opened to each other. I can.

According to various embodiments, when the signal transmitted from the controller is a fifth signal, the switch may electrically connect one end of the capacitor to the input terminal of the integrator and ground the other end of the capacitor.

According to various embodiments, the switch may ground the capacitor and the output terminal of the integrator when the signal transmitted from the controller is a sixth signal.

According to various embodiments, the artificial neural network model may include a convolution neural network (CNN) or a binarized neural network (BNN).

An electronic device according to various embodiments of the present disclosure includes a batch normalizing device for normalizing an artificial neural network model, and the batch normalizing device is a resistive memory array for convolutional processing a first analog signal transmitted from a first layer among the plurality of layers. (RRAM array), a circuit arrangement including a plurality of active elements for transmitting a third analog signal to a second layer by normalizing a second analog signal subjected to convolution from the resistive memory array, and a control unit configured to control the circuit arrangement It may include.

The methods according to the embodiments described in the claims or the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. One or more programs stored in a computer-readable storage medium are configured to be executable by one or more processors in an electronic device (device). The one or more programs include instructions for causing the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

These programs (software modules, software) include random access memory, non-volatile memory including flash memory, read only memory (ROM), and electrically erasable programmable ROM. (EEPROM: Electrically Erasable Programmable Read Only Memory), magnetic disc storage device, Compact Disc-ROM (CD-ROM), Digital Versatile Discs (DVDs), or other types of It may be stored in an optical storage device or a magnetic cassette. Alternatively, it may be stored in a memory composed of a combination of some or all of them. In addition, a plurality of configuration memories may be included.

In addition, the program is through a communication network composed of a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. It may be stored in an accessible storage device. Such a storage device may access a device performing an embodiment of the present disclosure through an external port. In addition, a separate storage device on the communication network may access a device performing an embodiment of the present disclosure.

In the above-described specific embodiments of the present disclosure, components included in the disclosure are expressed in the singular or plural according to the presented specific embodiments. However, the singular or plural expression is selected appropriately for the situation presented for convenience of description, and the present disclosure is not limited to the singular or plural constituent elements, and even constituent elements expressed in plural are composed of the singular or Even the expressed constituent elements may be composed of pluralities.

Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, various modifications may be made without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure is limited to the described embodiments and should not be defined, and should be determined by the scope of the claims and equivalents as well as the scope of the claims to be described later.

100: input
120: processor
130: memory
131: artificial neural network model
140: running processor
230: kernel matrix or resistive memory array
410: normalization circuit arrangement
420: control unit

Claims (13)

An input unit for inputting training data for training an artificial neural network model; And
Including a learning processor for training the artificial neural network model using the training data,
The artificial neural network model,
A plurality of layers receiving a signal corresponding to the data received from the input unit, and at least one batch normalization device disposed between the plurality of layers,
The arrangement normalization device normalizes a resistive memory array (RRAM array) for convolutional processing a first analog signal transmitted from a first layer among the plurality of layers, and a second analog signal convolutional processing from the resistive memory array. A neural network learning apparatus comprising a circuit arrangement including a plurality of switches and a plurality of electrical elements for transmitting a third analog signal to a second layer.
The method of claim 1,
The circuit arrangement,
Neural network training apparatus for controlling a voltage value transmitted through each channel of the resistive memory array.
The method of claim 1,
The batch normalization device,
And a control unit for controlling the circuit arrangement to normalize signals applied to each channel of the resistive memory arrangement.
The method of claim 3,
The control unit,
Neural network learning apparatus for transmitting a control signal for a switch operation of the circuit arrangement to the circuit arrangement.
The method of claim 4,
The plurality of electric elements of the circuit arrangement includes a plurality of integrators and a plurality of capacitors,
The switch is a neural network learning apparatus for changing an electrical path between an integrator and a capacitor based on the control signal of the controller.
The method of claim 5,
In the circuit arrangement, a plurality of circuits corresponding to a channel receiving an output second analog signal or a third analog signal are connected in parallel,
The capacitor is a neural network learning device connected to an input terminal or an output terminal of the integrator.
The method of claim 6,
The switch electrically connects the capacitor to the output terminal of the integrator when the signal transmitted from the control unit is a first signal, and a plurality of circuits are short-circuited to each other.
The method of claim 6,
The switch electrically connects one end of the capacitor to the input terminal of the integrator, and a plurality of circuits are connected in parallel when the signal transmitted from the controller is a second signal.
The method of claim 6,
The switch is a neural network learning apparatus for electrically connecting one end of the capacitor to an input terminal of the integrator and grounding the other end of the capacitor when the signal transmitted from the controller is a third signal.
The method of claim 9,
When the signal transmitted from the controller is a fourth signal, the switch electrically connects the other end of the capacitor to the input terminal of the integrator, grounds one end of the capacitor, and opens a plurality of circuits to each other.
The method of claim 10,
The switch is a neural network learning apparatus for electrically connecting one end of the capacitor to an input terminal of the integrator and grounding the other end of the capacitor if the signal transmitted from the controller is a fifth signal.
The method of claim 6,
The switch is a neural network learning device for grounding the capacitor and the output terminal of the integrator when the signal transmitted from the controller is a sixth signal.
The method of claim 1,
The artificial neural network model,
A neural network training device including a convolution neural network (CNN) or a binarized neural network (BNN).

Images