KR102134339B1 - Method and Apparatus for Detecting Fault of Multi-Core in Multi-Layer Perceptron Structure with Dropout - Google Patents

Abstract

In the present embodiments, each core is compared through a comparison between an operation result value performed in an inactive streaming processor and an original operation result value in an activated streaming processor in which dropout is not applied in a dropout operation to solve overfitting. It provides a computing system capable of verifying whether an error has occurred.

Description

Method and Apparatus for Detecting Fault of Multi-Core in Multi-Layer Perceptron Structure with Dropout in Multi-layer Perceptron Structure with Dropout}

The technical field to which the present embodiment pertains relates to a method and apparatus for detecting an error in a computing core in a computing system in which multiple computing cores process data in parallel.

The contents described in this section merely provide background information for this embodiment, and do not constitute a prior art.

An artificial neural network composed of a plurality of artificial neurons repeatedly computes a large amount of data to learn the computational value of each neuron. The repetitive operation of such a large amount of data has a disadvantage that it takes a long time to learn in a CPU (Central Processing Unit) composed of a small number of computation units, and a GPU (Graphics Processing Unit) composed of a large number of computational units to overcome this. ) Is required.

As the effect of learning of an artificial neural network using a GPU was verified, learning of various types of artificial neural networks was performed through the GPU, but the GPU has poor means to secure reliability in the result of the computation.

Currently, it has a correction means such as Error Correction Code (ECC), which is a means for verifying errors in the memory input/output stage. ECC corrects errors in the process of inputting and outputting memory, but cannot verify and correct errors that occur during the calculation of a large number of computing cores.

The verification of errors occurring in the process of calculating individual cores was performed using the same method as DMR (Dual Modular Redundancy). However, when using DMR, there is a disadvantage in that the consumption resources are doubled. There is a delay in learning time in the learning of artificial neural networks through which there is a problem in terms of efficiency.

Korean Patent Publication No. 10-2016-0099587 (2016.08.22.)

In the embodiments of the present invention, each result is compared with the result of an operation performed in an inactive streaming processor and an original operation result in an activated streaming processor in which a dropout is not applied in a dropout operation to solve overfitting. The main purpose of the invention is to verify whether an error has occurred in the core.

Still other unspecified objects of the present invention may be further considered within a range that can be easily deduced from the following detailed description and its effects.

According to an aspect of the present embodiment, in a method of detecting an error of an operation core in a multi-layer perceptron structure in a computing system in which multiple operation cores process data in parallel, a graphics processing unit (GPU) drops out at a predetermined rate. When (Drop Out) is performed, among the multiple operation cores included in the graphic processing unit, an operation defined in accordance with the multi-layer perceptron structure is performed in an activated operation core block to which the dropout is not applied, and a first result value is output. , Performing a calculation defined according to the multi-layer perceptron structure in an inactive calculation core block to which the dropout is applied among the multiple calculation cores to output a second result value, a central processing unit connected to the graphic processing unit (Central Processing Unit) , CPU) provides an error detection method of an operation core including comparing the first result value with the second result value and detecting an error operation core among the operation cores included in the graphic processing unit.

According to another aspect of the present embodiment, in a computing system in which multiple computation cores process data in parallel, when a dropout is performed at a predetermined ratio, an activation operation in which the dropout is not applied among the multiple computation cores The core block performs an operation defined according to the multi-layer perceptron structure, outputs a first result value, and performs an operation defined according to the multi-layer perceptron structure in an inactive operation core block in which the dropout is applied among the multi-operation cores. A graphics processing unit outputting a second result value, and connected to the graphic processing unit, comparing the first result value with the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit A computing system for parallel processing of data including a central processing unit is provided.

As described above, according to embodiments of the present invention, in the dropout operation for solving the overfitting, the result of the original operation performed in the activated streaming processor without dropout and the operation performed in the inactive streaming processor It is effective to verify whether an error has occurred in each core by comparing the result values.

Even if the effects are not explicitly mentioned here, the effects described in the following specification expected by the technical features of the present invention and the potential effects thereof are treated as described in the specification of the present invention.

1 is a block diagram illustrating a computing system according to an embodiment of the present invention.
2 is a diagram illustrating a multi-layer perceptron structure applied to a computing system according to an embodiment of the present invention.
3 is a diagram illustrating the operation of multiple computing cores of a computing system according to an embodiment of the present invention.
4 is a flowchart illustrating a method of detecting an error of an operational core according to another embodiment of the present invention.
5 is a graph illustrating an error detection rate for a dropout rate performed according to embodiments of the present invention.

Hereinafter, when it is determined that the subject matter of the present invention may be unnecessarily obscured by those skilled in the art with respect to known functions related to the present invention, the detailed description will be omitted, and some embodiments of the present invention will be omitted. It will be described in detail through exemplary drawings.

1 is a block diagram illustrating a computing system. The computing system fails in each core by comparing the result of the operation result from the deactivated streaming processor with the original operation result value from the active streaming processor without dropout in the dropout operation to solve the overfitting. Verify that has occurred. The computing system detects errors that may occur in the learning process of the artificial neural network using the GPU, can identify the cause of malfunctions and learning time delays that may occur in the artificial neural network according to the error, and builds a highly reliable artificial neural network. It is possible.

Referring to FIG. 1, the computing system 10 includes a central processing unit (CPU, 100) and a graphics processing unit (GPU, 200). The computing system 10 may omit some of the various components illustrated by example in FIG. 1 or additionally include other components. For example, the computing system 10 may further include a storage unit or a North Bridge. Northbridge is an integrated circuit that controls high-speed devices such as CPU, memory, BIOS, GPU, and Southbridge by bus. Southbridge controls the data flow of peripheral devices or manages power.

The central processing unit (CPU, 100) receives information from outside, stores, interprets and interprets computer program instructions, and outputs them to the outside. In other words, the central processing unit controls the operation of the entire computer while exchanging information with computer parts.

The graphics processing unit (GPU, 200) not only performs calculations for computer graphics, but can also be used to calculate applications. Programmable layers and high-precision operations can be connected to the graphics pipeline to perform stream processing on data. The graphics processing unit runs one kernel at a time in parallel on many records in the flow. A flow is simply a collection of records that require similar computation, and data parallelism can be implemented with flow. The kernel is a function applied to each element in the flow. The graphics processing unit contains a large number of cores and has memory inside. The plurality of graphic processing units are connected to the central processing unit 100 or the Northbridge.

The graphic processing unit 200 includes a plurality of streaming multiprocessors (SMs). The graphics processing unit 200 may include a streaming multiprocessor control unit controlling operations of a plurality of streaming multiprocessors. Each streaming multiprocessor (SM) has an independent instruction scheduler that can run multiple threads simultaneously. Streaming Processor (SP) performs basic logical and mathematical operations. SFU (Special Function Unit) is used for operations such as transcendental function and pixel property interpolation, and also includes floating point multiplier. When multiple threads are executed simultaneously in a streaming multiprocessor (SM), the same instruction (Broadcasting) is delivered to a predetermined number of SPs and a predetermined number of SFUs, where each unit (SP or SFU) sends the same instruction. However, registers and memory addresses are managed differently. Shared memory (Shared Memory) has a predetermined capacity and performs data exchange between threads running in a streaming multiprocessor. LD/ST performs a read command or a write command.

When the graphic processing unit 200 performs dropout at a predetermined rate, the first result is performed by performing a defined operation according to the multi-layer perceptron structure in an active computed core block to which dropout is not applied among multiple computed cores. Print the value. The graphic processing unit 200 performs a calculation defined according to the multi-layer perceptron structure in an inactive calculation core block to which the dropout is applied among multiple calculation cores, and outputs a second result value.

The central processing unit 100 is connected to the graphic processing unit 200 and compares the first result value and the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit.

2 is a diagram illustrating a multi-layer perceptron structure applied to a computing system.

The computing system compares the structure of the multi-layer perceptron and the result value derived from the deactivated core and the result value derived from the existing computational core.

In the structure of the multilayer perceptron, the weight calculation before being input to the activation function of neurons is expressed as in Equation 1.

y is the output value, x is the input value, w is the weight, and n is the number of neurons.

The multi-layer perceptron is calculated by multiplying the input value input to n artificial neurons and the weight represented by each artificial neuron, and delivering the result derived to the activation function to adjust and output the value by the activation function.

The multi-layer perceptron structure includes nodes of one or more layers connected by a network. The plurality of layers includes an input layer, a hidden layer, and an output layer, and each layer has a weight and learns the weight. Each layer may include parameters. The layer's parameters include a set of learnable filters. The parameters include weights and/or biases between nodes. Learning a plurality of parameters, some parameters can be shared. The layer of the neural network, the initial value of the weight, the bias to enter the node, the active function to be used at the node, the learning rate of the neural network, the loss function to measure the total error, the algorithm to minimize the error of the neural network, and the algorithm to regulate the over-adaptation depend on the design being implemented. Accordingly, suitable values and functions can be set.

When each layer of the artificial neural network calculates the output value of an artificial neuron, adjusts the weight through back propagation, and performs calculation, a problem of overfitting occurs. Accuracy in the learning process is high, but decreases accuracy in actual use. To solve this problem, a dropout technique has been proposed, and an example of an operation in which dropout is applied to an arbitrary neuron is expressed as Equation 2.

Equation 2 shows when a hidden layer has a dropout process with a probability of 50% in an artificial neural network composed of 4 neurons. When Equation 2 is developed, Equation 3 is derived and it is possible to improve the learning speed and prevent overfitting by reducing the computational load in the learning process. The rate of dropout may be set within a range from 0.1 to 0.9 based on the maximum value of 1, and when the rate of dropout is 0.5 or more, the error detection rate is 100%.

The computing system verifies errors that may occur when an operation is performed using the computation core of the GPU by performing the same operation of neurons adjacent to the operation of 0 and comparing the values. In a multi-layer perceptron structure, a node set not to perform an operation according to a dropout performs the same operation of an adjacent node.

The calculation using the neuron to which the computing system has a dropout is expressed as in Equation 4.

When Equation 4 is developed, Equation 5 is derived, and it can be confirmed that the equations in terms 1, 2, 3, and 4 of the matrix match. Generated by designating the number of corresponding nodes in a multi-layer perceptron structure as a block for a streaming processor (SP) sharing a control cache and a command cache in a streaming multiprocessor (SM) in a plurality of graphic processing units Threads perform operations on each streaming processor.

In order to substitute Equation (5) into the streaming processor of the GPU, the number of neurons is designated as a block corresponding to the number of streaming processors of the streaming multiprocessor group, so that the generated thread can perform operations on each streaming processor. It is verified whether an error has occurred in each core by comparing the result of an operation performed by a streaming processor inactive due to dropout and an original operation result performed by a streaming processor without dropout applied.

An example when the calculation of Equation 5 is performed on the GPU is shown in FIG. 3.

The streaming processor to which the dropout is not applied outputs a first result value, and the streaming processor to which the dropout is applied outputs a second result value. The computing system compares the first result value and the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit.

Although components included in the computing system are illustrated separately in FIG. 1, a plurality of components may be combined with each other and implemented as at least one module. The components are connected to a communication path connecting a software module or a hardware module inside the device to operate organically with each other. These components communicate using one or more communication buses or signal lines.

The computing system may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general purpose or special purpose computer. The device may be implemented using a fixed-wired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a System on Chip (SoC) including one or more processors and controllers.

The computing system may be mounted on a computing device provided with hardware elements in the form of software, hardware, or a combination thereof. Computing devices include various devices or communication devices such as communication modems for performing communication with wired/wireless communication networks, memory for storing data for executing programs, and microprocessors for executing and calculating and executing programs. It can mean a device.

4 is a flowchart illustrating a method of detecting an error of an operational core according to another embodiment of the present invention. The error detection method of the computational core may be performed by a computing system. Detailed descriptions of operations performed by the computing system and overlapping descriptions will be omitted.

In step S410, if the graphics processing unit (Graphics Processing Unit, GPU) performs a drop-out at a predetermined rate, among the multiple computation cores included in the graphics processing unit, the active computation core block to which the dropout is not applied is applied. Perform the operation defined according to the multi-layer perceptron structure to output the first result value, and perform the operation defined according to the multi-layer perceptron structure in the inactive operation core block with dropout among multiple operation cores to output the second result value. . The streaming processor to which the dropout is not applied outputs a first result value, and the streaming processor to which the dropout is applied outputs a second result value.

In step S420, the central processing unit (CPU) connected to the graphic processing unit compares the first result value and the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit.

FIG. 5 is a graph showing an error detection rate when MNIST, which is a representative data set for learning using a multi-layer perceptron using a GPU, is set with different drop-out ratios. When the dropout rate is 50% or more, an inactive streaming processor may include all active streaming processors, and thus, an error detection rate is 100%.

According to the present embodiments, it is possible to check for errors occurring in the operation of the multi-layer percept using the GPU, and it is possible to identify the cause of the non-ideal execution time occurring in the learning process. It is possible to secure the reliability of the learning results for the artificial neural network, and more accurate design of the artificial neural network is possible.

Although FIG. 4 describes that each process is executed sequentially, this is merely an example, and a person skilled in the art changes and executes the sequence described in FIG. 4 without departing from the essential characteristics of the embodiments of the present invention. Or, it may be applied by various modifications and variations by executing one or more processes in parallel or adding other processes.

The operation according to the present embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. Computer readable media refers to any media that participates in providing instructions to a processor for execution. Computer-readable media may include program instructions, data files, data structures, or combinations thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. The computer program may be distributed over a networked computer system to store and execute computer readable code in a distributed manner. Functional programs, codes, and code segments for implementing the present embodiment can be easily inferred by programmers in the technical field to which this embodiment belongs.

The present embodiments are for explaining the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the claims below, and all technical spirits within the equivalent range should be interpreted as being included in the scope of the present embodiment.

10: computing system
100: central processing unit
200: graphics processing unit

Claims (10)

In a computing system in which multiple computing cores process data in parallel, in a multi-layer perceptron structure, an error detection method of a computing core includes:
When a graphics processing unit (GPU) performs dropout at a predetermined rate, among the multiple computing cores included in the graphics processing unit, the multi-core in the active computing core block to which the dropout is not applied is multi-layered. Perform the operation defined according to the perceptron structure to output the first result value, and perform the operation defined according to the multi-layer perceptron structure in the inactive operation core block to which the dropout is applied among the multiple operation cores to obtain the second result value. Outputting;
The central processing unit (CPU) connected to the graphic processing unit compares the first result value with the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit. Includes,
The multilayer perceptron structure includes nodes of a plurality of layers connected by a network,
In the multi-layer perceptron structure, a node set not to perform an operation according to the dropout performs the operation of an adjacent node identically, and the central processing unit performs the same operation to perform the operation of the first result value and the second result. If the values are the same, no error has occurred, and if the first result value and the second result value are not the same, it is determined that an error has occurred. According to claim 1,
The ratio related to the dropout is set within a range from 0.1 to 0.9 based on the maximum value 1, and when the ratio related to the dropout is 0.5 or more, an error detection rate of 100% is calculated, characterized in that the error detection rate is 100% Way. According to claim 1,
The plurality of layers includes an input layer, a concealment layer, and an output layer, and each layer has a weight and learns the weight. According to claim 1,
Generated by designating the number of corresponding nodes in the multi-layer perceptron structure as a block for a streaming processor (SP) that shares the control logic and instruction cache in a streaming multiprocessor (SM) in the graphic processing unit. A method for detecting an error in an operation core, characterized in that a thread performs an operation in each streaming processor. According to claim 4,
The streaming processor to which the dropout is not applied outputs the first result value,
A method of detecting an error in an operational core, characterized in that the streaming processor to which the dropout is applied outputs the second result value. In a computing system in which multiple computing cores process data in parallel,
When a drop-out is performed at a preset ratio, an activated operation core block to which the dropout is not applied among the multiple operation cores performs a defined operation according to a multi-layer perceptron structure to output a first result value, A graphics processing unit performing a defined operation according to the multi-layer perceptron structure in an inactive computation core block to which the dropout is applied among the multiple computation cores to output a second result value; And
And a central processing unit connected to the graphic processing unit and comparing the first result value with the second result value to detect an error-producing operation core among the operation cores included in the graphic processing unit,
The multilayer perceptron structure includes nodes of a plurality of layers connected by a network,
In the multi-layer perceptron structure, a node set not to perform an operation according to the dropout performs the operation of an adjacent node identically, and the central processing unit performs the same operation to perform the operation of the first result value and the second result. If the values are the same, an error has not occurred, and if the first result value and the second result value are not the same, a computing system for parallel processing of data characterized by determining that an error has occurred. The method of claim 6,
The ratio related to the dropout is set within a range from 0.1 to 0.9 based on the maximum value of 1, and when the ratio related to the dropout is 0.5 or more, an error detection rate is 100% to process data in parallel. Computing system. The method of claim 6,
The plurality of layers includes an input layer, a concealment layer, and an output layer, and each layer has a weight and a computing system for parallel processing of data characterized by learning the weight. The method of claim 6,
Generated by designating the number of corresponding nodes in the multi-layer perceptron structure as a block for a streaming processor (SP) that shares the control logic and instruction cache in a streaming multiprocessor (SM) in the graphic processing unit. A computing system for parallel processing of data characterized in that a thread performs an operation on each streaming processor. The method of claim 9,
The streaming processor to which the dropout is not applied outputs the first result value,
The streaming processor to which the dropout is applied is a computing system for processing data in parallel, characterized by outputting the second result value.

Images