KR20200041154A - Method and Apparatus for Detecting Fault of Multi-Core in Multi-Layer Perceptron Structure with Dropout - Google Patents

Abstract

According to embodiments of the present invention, provided is a computing system capable of verifying whether an error occurs in each core by comparing an original operation result value performed in an activated streaming processor to which a dropout technique is not applied and an operation result value performed in a deactivated streaming processor, in a dropout operation for solving overfitting.

Description

Method and Apparatus for Detecting Fault of Multi-Core in Multi-Layer Perceptron Structure with Dropout in Multi-layer Perceptron Structure with Dropout}

The technical field to which this embodiment pertains relates to a method and apparatus for detecting an error in a computing core in a computing system in which multiple computing cores process data in parallel.

The contents described in this section merely provide background information for this embodiment, and do not constitute a prior art.

An artificial neural network composed of a plurality of artificial neurons repeatedly calculates a large amount of data and learns the computational value of each neuron. The repetitive operation of such a large amount of data has a disadvantage that it takes a long time to learn in a CPU (Central Processing Unit) composed of a small number of computational units, and a GPU (Graphics Processing Unit) composed of a large number of computational units to overcome this. ) Is required.

As the effects of learning of an artificial neural network using a GPU have been verified, learning of various types of artificial neural networks has been performed through the GPU, but the GPU has poor means to secure reliability in the result of the computation.

Currently, it has a correction means such as Error Correction Code (ECC), which is a means for verifying errors in the memory input / output stage. ECC corrects errors in the process of inputting and outputting memory, but cannot verify and correct errors that occur during calculation of a large number of computing cores.

The verification of errors occurring in the process of calculating individual cores was performed using the same method as DMR (Dual Modular Redundancy). However, when using DMR, there is a disadvantage in that the consumption resources are doubled. In learning the artificial neural network through, a delay in learning time occurs, which is problematic in terms of efficiency.

Korean Patent Publication No. 10-2016-0099587 (2016.08.22.)

In the embodiments of the present invention, each of the results through comparison of the result of the operation performed in the deactivated streaming processor and the original operation result in the activated streaming processor without dropout applied in the dropout operation to solve the overfitting The main purpose of the invention is to verify whether an error has occurred in the core.

Still other unspecified objects of the present invention can be further considered within the scope of being easily deduced from the following detailed description and its effects.

According to an aspect of the present embodiment, in a method of detecting an error of an operation core in a multi-layer perceptron structure in a computing system in which multiple operation cores process data in parallel, a graphics processing unit (GPU) drops out at a predetermined rate. When (Drop Out) is performed, among the multiple operation cores included in the graphic processing unit, the operation result defined according to the multi-layer perceptron structure is performed on the activated operation core block to which the dropout is not applied, and a first result value is output. , Performing a defined operation according to the multi-layer perceptron structure in an inactive computation core block to which the dropout is applied among the multiple computation cores, and outputting a second result value, a central processing unit connected to the graphic processing unit (Central Processing Unit) , CPU) compares the first result value with the second result value Group provides a processing core among the failure operation core erroneous detection of the operation core comprising the steps included in the method of detecting a graphics processing unit.

According to another aspect of the present embodiment, in a computing system in which multiple computation cores process data in parallel, when a dropout is performed at a predetermined rate, an activation operation in which the dropout is not applied among the multiple computation cores is performed. The core block performs an operation defined according to the multi-layer perceptron structure, outputs a first result value, and performs an operation defined according to the multi-layer perceptron structure in an inactive operation core block in which the dropout is applied among the multiple operation cores. A graphic processing unit outputting a second result value, and connected to the graphic processing unit, comparing the first result value with the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit For computing parallel processing of data containing a central processing unit Provide system.

As described above, according to embodiments of the present invention, in the dropout operation for solving the overfitting, the result of the original operation performed in the activated streaming processor without dropout and the operation performed in the inactive streaming processor It is effective to verify whether an error has occurred in each core by comparing the result values.

Even if the effects are not explicitly mentioned here, the effects described in the following specification expected by the technical features of the present invention and the potential effects thereof are treated as described in the specification of the present invention.

1 is a block diagram illustrating a computing system according to an embodiment of the present invention.
2 is a diagram illustrating a multi-layer perceptron structure applied to a computing system according to an embodiment of the present invention.
3 is a diagram illustrating the operation of multiple computing cores of a computing system according to an embodiment of the present invention.
4 is a flowchart illustrating a method for detecting an error of an operational core according to another embodiment of the present invention.
5 is a graph illustrating an error detection rate for a dropout rate performed according to embodiments of the present invention.

Hereinafter, when it is determined that the subject matter of the present invention may be unnecessarily obscured by those skilled in the art with respect to known functions related to the present invention, the detailed description will be omitted, and some embodiments of the present invention will be omitted. It will be described in detail through exemplary drawings.

1 is a block diagram illustrating a computing system. The computing system fails at each core by comparing the result of the operation result from the deactivated streaming processor with the original operation result value from the active streaming processor without dropout in the dropout operation to solve the overfitting. Verify that has occurred. The computing system detects errors that may occur in the learning process of the artificial neural network using the GPU, can identify the cause of malfunctions and learning time delays that may occur in the artificial neural network due to errors, and builds an artificial network with high reliability. It is possible.

Referring to FIG. 1, the computing system 10 includes a central processing unit (CPU, 100) and a graphics processing unit (GPU, 200). The computing system 10 may omit some components or further include other components among various components illustrated by way of example in FIG. 1. For example, the computing system 10 may further include a storage unit or a North Bridge. Northbridge is an integrated circuit that controls and connects high-speed devices such as CPU, memory, BIOS, GPU, and Southbridge by bus. Southbridge controls the data flow of peripheral devices or manages power.

The central processing unit (CPU, 100) receives information from outside, stores, interprets and interprets computer program instructions, and outputs them to the outside. In other words, the central processing unit controls the overall operation of the computer while exchanging information with computer parts.

The graphics processing unit (GPU, 200) not only performs calculations for computer graphics, but can also be used to calculate applications. Stream processing can be performed on data by linking programmable layers and high-precision operations to the graphics pipeline. The graphics processing unit runs one kernel at a time in parallel on many records in the flow. A flow is simply a collection of records that require similar computation, and data parallelism can be implemented with flow. The kernel is a function applied to each element in the flow. The graphics processing unit contains a large number of cores and has memory therein. The plurality of graphic processing units are connected to the central processing unit 100 or the Northbridge.

The graphic processing unit 200 includes a plurality of streaming multiprocessors (SMs). The graphics processing unit 200 may include a streaming multiprocessor control unit controlling operations of a plurality of streaming multiprocessors. Each streaming multiprocessor (SM) has an independent instruction scheduler that can run multiple threads simultaneously. Streaming Processor (SP) performs basic logical and mathematical operations. SFU (Special Function Unit) is used for operations such as transcendental function and pixel property interpolation, and it also includes floating point multiplier. When multiple threads are executed simultaneously in a streaming multiprocessor (SM), the same instruction (Broadcasting) is delivered to a predetermined number of SPs and a predetermined number of SFUs, where each unit (SP or SFU) sends the same instruction. However, registers and memory addresses are managed differently. Shared memory has a predetermined capacity and performs data exchange between threads running in a streaming multiprocessor. LD / ST performs a read command or a write command.

When the graphics processing unit 200 performs dropout at a predetermined rate, the first result is performed by performing a calculation defined according to a multi-layer perceptron structure in an active computed core block to which dropout is not applied among multiple computed cores. Print the value. The graphic processing unit 200 performs a defined operation according to the multi-layer perceptron structure in an inactive operation core block to which the dropout is applied among multiple operation cores, and outputs a second result value.

The central processing unit 100 is connected to the graphic processing unit 200 and compares the first result value and the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit.

2 is a diagram illustrating a multi-layer perceptron structure applied to a computing system.

The computing system compares the structure of the multi-layer perceptron and the result value derived from the deactivated core and the result value derived from the existing computational core.

In the structure of the multi-layer perceptron, the weight calculation before being input to the activation function of neurons is expressed as Equation 1.

y is the output value, x is the input value, w is the weight, and n is the number of neurons.

The multi-layer perceptron is calculated by multiplying the input value input to n artificial neurons and the weight represented by each artificial neuron, and passing the result derived to the activation function to adjust and output the value by the activation function.

The multilayer perceptron structure includes nodes of one or more layers connected by a network. The plurality of layers includes an input layer, a hidden layer, and an output layer, and each layer has a weight and learns weight. Each layer can include parameters. The layer's parameters include a set of learnable filters. The parameters include weights and / or biases between nodes. Learning a plurality of parameters, some parameters can be shared. The layer of the neural network, the initial value of the weight, the bias to enter the node, the active function to be used at the node, the learning rate of the neural network, the loss function to measure the total error, the algorithm to minimize the error of the neural network, and the algorithm to regulate the over-adaptation depend on the design being implemented. Accordingly, suitable values and functions can be set.

When each layer of an artificial neural network calculates the output value of an artificial neuron, adjusts the weight through back propagation, and performs calculation, a problem of overfitting occurs. Accuracy in the learning process is high, but decreases accuracy in actual use. To solve this problem, a dropout technique has been proposed, and an example of an operation in which dropout is applied to an arbitrary neuron is expressed as Equation 2.

Equation 2 shows a case where a hidden layer has a dropout process with a probability of 50% in an artificial neural network composed of 4 neurons. When Equation 2 is developed, Equation 3 is derived, and it is possible to improve the learning speed and prevent overfitting by reducing the computational load in the learning process. The ratio of the dropout may be set within a range of 0.1 to 0.9 based on the maximum value 1, and when the ratio of the dropout is 0.5 or more, the error detection rate is 100%.

The computing system verifies errors that may occur when an operation is performed using a computation core of a GPU by performing the same operation of neurons adjacent to the operation of 0 and comparing the values. In a multi-layer perceptron structure, nodes configured not to perform operations according to dropouts perform operations of adjacent nodes in the same way.

The calculation using the neuron to which the computing system has applied dropout is expressed as in Equation 4.

When Equation 4 is developed, Equation 5 is derived, and it can be confirmed that the equations in terms 1, 2, 3, and 4 of the matrix match. In a multi-layer perceptron structure, a number of corresponding nodes in a multi-layer perceptron structure is generated in blocks for a streaming processor (SP) that shares control caches and control caches in a streaming multiprocessor (SM) in a plurality of graphic processing units. Threads perform operations on each streaming processor.

In order to substitute Equation 5 into the streaming processor of the GPU, the number of neurons is designated as a block corresponding to the number of streaming processors of the streaming multiprocessor group, so that the generated thread can perform operations on each streaming processor. It is verified whether an error has occurred in each core by comparing the result of an operation performed by a streaming processor inactive due to dropout and an original operation result performed by a streaming processor without dropout applied.

An example when the calculation of Equation 5 is performed on the GPU is shown in FIG. 3.

The streaming processor to which the dropout is not applied outputs the first result, and the streaming processor to which the dropout is applied outputs the second result. The computing system compares the first result value and the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit.

Although components included in the computing system are illustrated separately in FIG. 1, a plurality of components may be combined with each other and implemented as at least one module. The components are connected to a communication path connecting a software module or hardware module inside the device to operate organically with each other. These components communicate using one or more communication buses or signal lines.

The computing system may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general purpose or special purpose computer. The device may be implemented using a fixed-wired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a System on Chip (SoC) including one or more processors and controllers.

The computing system may be mounted on a computing device provided with hardware elements in the form of software, hardware, or a combination thereof. Computing devices include various devices or communication devices such as communication modems for performing communication with wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing and calculating and executing programs. It can mean a device.

4 is a flowchart illustrating a method for detecting an error of an operational core according to another embodiment of the present invention. The error detection method of the computational core may be performed by a computing system. Detailed descriptions of operations performed by the computing system and overlapping descriptions will be omitted.

In step S410, if the graphics processing unit (Graphics Processing Unit, GPU) performs a dropout (Drop Out) at a predetermined rate, among the multiple calculation cores included in the graphics processing unit, the active calculation core block to which the dropout is not applied is Perform the operation defined according to the multi-layer perceptron structure to output the first result value, and perform the operation defined according to the multi-layer perceptron structure in the inactive operation core block with dropout applied among the multiple operation cores to output the second result value. . The streaming processor to which the dropout is not applied outputs the first result, and the streaming processor to which the dropout is applied outputs the second result.

In step S420, the central processing unit (CPU) connected to the graphic processing unit compares the first result value and the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit.

FIG. 5 is a graph showing an error detection rate when MNIST, which is a representative data set for learning using a multi-layer perceptron using a GPU, is set with different drop-out ratios. When the dropout rate is 50% or more, an inactive streaming processor may include all active streaming processors, and thus, an error detection rate is 100%.

According to the present embodiments, it is possible to check errors occurring in the operation of the multi-layer percept using the GPU, and it is possible to identify the cause of the non-ideal execution time occurring in the learning process. It is possible to secure the reliability of the learning results for the artificial neural network, and more accurate design of the artificial neural network is possible.

Although FIG. 4 describes that each process is executed sequentially, this is merely illustrative, and a person skilled in the art changes and executes the sequence described in FIG. 4 without departing from the essential characteristics of the embodiments of the present invention. Or, it may be applied by various modifications and variations by executing one or more processes in parallel or adding other processes.

The operation according to the present embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. Computer readable media refers to any media that participates in providing instructions to a processor for execution. Computer-readable media may include program instructions, data files, data structures, or combinations thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. The computer program may be distributed over a networked computer system to store and execute computer readable code in a distributed manner. Functional programs, codes, and code segments for implementing the present embodiment can be easily inferred by programmers in the technical field to which this embodiment belongs.

The present embodiments are for explaining the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these examples. The protection scope of the present embodiment should be interpreted by the claims below, and all technical spirits within the equivalent range should be interpreted as being included in the scope of the present embodiment.

10: computing system
100: central processing unit
200: graphics processing unit

Claims (10)

In a computing system in which multiple computing cores process data in parallel, in a multi-layer perceptron structure, an error detection method of a computing core includes:
If the graphics processing unit (Graphics Processing Unit, GPU) performs dropout at a predetermined rate, among the multiple computational cores included in the graphics processing unit, the multi-core in the active computational core block to which the dropout is not applied is multi-layered. Perform the operation defined according to the perceptron structure to output the first result value, and perform the operation defined according to the multi-layer perceptron structure in the inactive operation core block to which the dropout is applied among the multiple operation cores to obtain the second result value. Outputting;
A central processing unit (CPU) connected to the graphic processing unit compares the first result value and the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit.
Method of error detection of the operation core comprising a. According to claim 1,
The ratio related to the dropout is set within a range from 0.1 to 0.9 based on the maximum value of 1, and when the ratio related to the dropout is 0.5 or more, an error detection rate of 100% is calculated, characterized in that the error detection rate is 100%. Way. According to claim 1,
The multi-layer perceptron structure includes nodes of a plurality of layers connected by a network, the plurality of layers includes an input layer, a hidden layer, and an output layer, each layer having a weight and learning the weight Error detection method According to claim 1,
In the multi-layer perceptron structure, a node configured not to perform an operation according to the dropout performs the same operation of an adjacent node,
Generated by designating the number of corresponding nodes in the multi-layer perceptron structure as a block for a streaming processor (SP) sharing a control logic and an instruction cache in a streaming multiprocessor (SM) in the graphic processing unit A method for detecting an error in an operation core, characterized in that a thread performs an operation in each streaming processor. According to claim 4,
The streaming processor to which the dropout is not applied outputs the first result value,
The method for detecting an error in an operational core, characterized in that the streaming processor to which the dropout is applied outputs the second result value. In a computing system in which multiple computing cores process data in parallel,
When a drop-out is performed at a preset ratio, an active operation core block to which the drop-out is not applied among the multiple operation cores performs a defined operation according to the multi-layer perceptron structure to output a first result value, A graphics processing unit performing a calculation defined according to the multi-layer perceptron structure in an inactive calculation core block to which the dropout is applied among the multiple calculation cores to output a second result value; And
A central processing unit connected to the graphic processing unit to detect an error-producing operation core among operation cores included in the graphic processing unit by comparing the first result value and the second result value
Computing system for parallel processing of data comprising a. The method of claim 6,
The ratio related to the dropout is set within a range from 0.1 to 0.9 based on the maximum value of 1, and when the ratio related to the dropout is 0.5 or more, an error detection rate is 100% to process data in parallel. Computing system. The method of claim 6,
The multi-layer perceptron structure includes nodes of a plurality of layers connected by a network, the plurality of layers includes an input layer, a hidden layer, and an output layer, each layer having a weight and learning the weight A computing system that processes data in parallel. The method of claim 6,
In the multi-layer perceptron structure, a node configured not to perform an operation according to the dropout performs the same operation of an adjacent node,
By designating the number of corresponding nodes in the multi-layer perceptron structure as a block for a streaming processor (SP) that shares a control cache and a command cache in a streaming multiprocessor (SM) in the plurality of graphic processing units, as a block A computing system for parallel processing of data characterized in that the created thread performs an operation on each streaming processor. The method of claim 9,
The streaming processor to which the dropout is not applied outputs the first result value,
The streaming processor to which the dropout is applied is a computing system for processing data in parallel, characterized by outputting the second result value.

Images