JP6709266B2 - Processing in neural networks - Google Patents

Description

The present disclosure relates to processing data in neural networks.

Neural networks are used in the fields of machine learning and artificial intelligence. A neural network comprises a set of node sets, the nodes being interconnected by links and interacting with each other. The principle of neural networks in computing is based on information about how electrical stimuli convey information in the human brain. For this reason, nodes are often called neurons. It is also sometimes called the apex. Links are sometimes called edges. The network can receive input data and a particular node operates on the data. The results of these operations are passed to other nodes. The output of each node is called the node liveness value or node value. Each link is associated with a weight. The weight defines the connectivity between the nodes of the neural network. Many different techniques are known which allow a neural network to learn by varying the value of the weights.

FIG. 1A shows a highly simplified form of a node configuration in a neural network. This type of configuration is often used in learning or training and comprises an input layer of nodes, a hidden layer of nodes, and an output layer of nodes. In reality there are many nodes in each layer and now there can be multiple layers per section. Each node of the input layer Ni is capable of producing an active or node value at its output, the active or node value being produced by applying a function to the data provided to that node. It The vector of node values from the input layer is scaled by the respective weight vector at the input of each node in the hidden layer. Each weight defines the connectivity between that particular node in the input layer and the node in the hidden layer to which it is connected. In practice, networks have millions of nodes and may be connected in multiple dimensions, so vectors are often tensors. The weights applied to the inputs of the node Nh are represented by w ₀ ,..., W ₂ . Each node in the input layer is at least initially connected to each node in the hidden layer. Each node in the hidden layer can apply an activation function to the data provided to them, as well as generate an output vector that is fed to each node N ₀ in the output layer. . Each node weights its input data by, for example, calculating the dot product of the input activation value of the node and its unique weight for each input link. An activation function is then applied to the weighted data. The activation function may be, for example, a sigmoid function. See FIG. 1B. The network processes the data input to the input layer, assigns weights to the activation values from each node, and acts on the data input to each node in the hidden layer (weighting nodes to activation function). Learn by applying. Therefore, the nodes in the hidden layer process the data and provide the output to the nodes in the output layer. Output layer nodes can also assign weights to their input data. Each weight is characterized by a respective error value. Further, each node may be associated with an error state. The error state at each node provides a measure of whether the error in the weight of the node is below a certain tolerance level or tolerance. There are various learning techniques, but in each case forward propagation through the network from left to right in FIG. 1A, calculation of the overall error, and error from right to left in FIG. 1A through the network. There is back propagation. In the next cycle, each node takes into account the backpropagated error and produces a revised set of weights. In this way, the network can be trained to perform its desired behavior.

One problem that can occur with neural networks is "overfitting." Large networks with millions or billions of parameters (weights) are prone to overfitting. By overfitting, rather than having the trained neural net be trained to extract the relevant features from the sample more generally to suit the application of extracting the features, the network was provided to it. Remember each training sample (the training sample that provides the data to the input node). Extensive techniques have been developed to solve this problem by ordering neural networks to avoid over-learning/over-memory.

A technique called "drop out" is discussed in JE Hinton et al., "Improving Neural Networks by Presenting Co-Adaptation of Feature Detectors" CoRR abs/1207580 (2012). According to this technique, for each training example, forward propagation involves randomly removing half of the activations at each layer. This error is then propagated back through the remaining activations only. This has been shown to significantly reduce overfitting and improve network performance.

Another technique known as "drop connect" is discussed in the paper "Regulation of Neural Network using Drop connect" by Li Wan et al., published by Department of Computer Science NYU. According to this technique, for each training example, some of the weights are randomly zeroed during forward propagation. This technique has also been shown to improve the performance of neural networks as well.

There are various challenges in implementing neural networks using known computer techniques. For example, it is not easy to implement a specific technique such as a dropout or a drop connection using a CPU or GPU, and if this is achieved, there is a complete benefit that an efficient implementation can be achieved.

The inventors have developed an execution unit for a processor that can efficiently implement a dropout or drop connection based on a single instruction in the instruction sequence of the processing unit.

According to one aspect of the invention, an execution unit for executing a computer program including a sequence of instructions, the sequence including masking instructions, the execution unit configured to execute the masking instructions, the masking instructions Masks values randomly selected from source operands with n values and fixes other original values from the source operands when executed by the execution unit to fix the original values from the source operands. An execution unit is provided that produces a result that includes a value and a masked value in each original position.

The execution unit may comprise a hardware pseudo-random number generator (HPRNG) configured to generate a randomized bit string from which a random number for randomly selecting a masked value. A bit sequence is derived.

In one embodiment, each randomized bit string output from HPRNG includes m bits, and the execution unit divides the m bits into p sequences and compares each sequence with a probability value to select a value. Configured to generate a weighted probability indicator for dynamically masking.

The masking instruction may include a source field that identifies a source operand, an indication of a destination register to hold the result, and a probability field that defines a probability value.

The execution unit may comprise an input buffer for holding source operands and an output buffer for holding results.

The value of the source operand may represent the weight of a link in the neural network (eg, to implement a drop connection).

The value in the source operand may represent an active value that defines the output value of a node in the neural network (eg, to implement dropout).

The sequence of instructions may include instructions for performing a dot product calculation of the result with the further set of values, and/or for writing the result to a memory location after the masking instruction has been executed.

The source operand may include any suitable number with any suitable length, for example (as a non-limiting example) four 16-bit values, two 32-bit values, or eight 8-bit values. Contains the value. In general, the n values may be one or more values.

Corresponding methods and computer programs are provided.

One aspect is a method of executing a computer program that includes a sequence of instructions, the sequence including a masking instruction, the method randomizing a value from a source operand having n values in response to executing the masking instruction. And fixing other original values from the source operand to produce a result that includes the original value from the source operand and the masked value at each original position. ..

A randomized bit string output including m bits may be provided, the method for dividing the m bits into p sequences and comparing each sequence with a probability value to selectively mask the values. It may include generating a weighted probability indicator.

The masking instruction may include a source field that identifies a source vector, an indication of a destination register to hold the result, and a probability field that defines a probability value.

The sequence of instructions may include instructions for performing a dot product calculation of the result with the further set of values.

The sequence of instructions may include instructions for writing the result to a memory location after the masking instruction has been executed.

Another aspect is a computer program that includes a sequence of computer readable instructions stored on a non-transmission medium, the sequence including masking instructions, the masking instructions having n values when executed. A result that randomly selects a value from an operand and masks it, fixing other original values from the source operand and including the original value from the source operand and the masked value at each original position To provide a computer program.

As used herein, the terms "random" and "randomly" shall mean random or pseudo-random.

A very simplified schematic of a neural network. A very simplified schematic of a neuron. 3 is a schematic view of a processing unit according to an embodiment of the present invention. FIG. The figure which shows the format of a masking instruction. FIG. 6 is a block diagram of an execution unit for implementing masking instructions.

FIG. 2 shows a schematic block diagram of an execution unit configured to execute a single instruction for masking randomly selected values in a vector. In this specification, this instruction is called an rmask instruction. The execution unit 2 forms a part of the pipeline 4 in the processing unit. The processing unit comprises an instruction fetch unit 6 that fetches instructions from the instruction memory 10. The processing unit also comprises a memory access stage 8 responsible for accessing the data memory 12 for loading data from the data memory 12 or for storing data in the memory. A set of registers 14 is provided to hold source and destination operands for the instructions executed by pipeline 4 in any case. It will be readily appreciated that the pipeline 4 may include many different types of execution units for executing a variety of different instructions, for example performing mathematical operations. One type of processing unit that may be useful in the present invention is a processing unit that uses barrel thread timeslots, in which supervisor threads have different worker threads different for their execution. Can be assigned to time slots. The rmask instructions described herein can be used with any suitable processing unit architecture. FIG. 3 shows the rmask instruction format. The instruction has a field 30 that defines the source memory location where the source operand is retrieved and a field 32 that relates to the destination memory location where the output vector is located. The memory location may be an implicitly or explicitly defined register in register 14, or an address in data memory. The rmask instruction further defines the probability value Src1 34, and the individual values in the source operand are maintained along with the probability value Src1 34, as described in more detail below. The instruction has an opcode 36, which identifies the instruction as an rmask instruction. There are two different forms of the rmask instruction. In one form of the instruction, the source operand may represent a vector containing four 16-bit values (half-precision vector), and in another form the source operand contains two 32-bit values (single-precision vector). May define a containing vector.

In either case, the rmask instruction has the effect of masking (zeroing) randomly selected values in the source vector. Such features have several different uses, and include among others the features described above known in the field of neural networks as "drop connections" and "dropouts". As already mentioned, neural networks can provide vectors representing weights, which represent the links between neurons in the net. Another vector can represent activity values, which are the values output to the individual neurons connected by the link.

A very common function in neural networks is to compute the dot product between these two vectors. For example, the dot product may be the pre-activation value, which is the value from which the neuron output is calculated using the activation function. Prior to calculating this dot product, application of the rmask instruction to the vector of weights can be used to implement the drop connection function. Applying the rmask instruction to a vector of weights randomly zeros the individual weights in the vector.

Applying the rmask instruction to a vector of liveness values before computing the dot product can be used to implement the dropout function. Applying the rmask instruction to a vector of liveness values will randomly zero each liveness value in the vector.

Vectors can represent floating point numbers in single or half precision. One type of vector can contain four 16-bit values (half-precision vector), and another type of vector can contain two 32-bit values (single-precision vector). FIG. 2 shows an execution unit operating on four 16-bit values or two 32-bit values. The execution unit 2 in FIG. 2 has an input buffer 20, which can hold four 16-bit values or two 32-bit values. It is shown to hold four 16-bit values respectively representing the weights w ₀ ,..., W ₃ . This unit comprises a hardware-implemented pseudo-random number generator 22 for providing a random number to the rmask module 24. An input buffer location 26 is provided to maintain the probability that each individual value in the vector will be maintained. This probability value can be provided within the instruction (as shown in FIG. 3) or set by a previous instruction to access the register or memory address by the rmask instruction. For example, this probability value may be a 17-bit immediate value in the information. The rmask module uses the random numbers generated by PRNG 22 and probability value 26 to randomly mask (zero) the values of the input vector. The output vector is put into the output buffer 28. In FIG. 2, the first and third values are shown as masked to zero. Note that this probability is the probability that each individual value will be maintained (ie unmasked). For example, if the probability is 0.5, then the probability that each value is masked is 0.5. Therefore, the probability that all four values are masked to 0 is (0.5) ⁴ , or 1/16. The random number generator introduces the required randomness into the masking by providing an indication of which value should be masked based on the probability.

Referring to FIG. 4, the rmask module operates as follows. The pseudo random number generator 22 generates a 64-bit output. For each output, four 16-bit fields cf0,...,cf3 are generated and tested against the probability value in the source operand. Note that for single precision mode, two 32-bit fields are generated and likewise tested against the probability value in the source operand.

A 64-bit output of PRNG is provided when the mask instruction is executed. For each output, four 16-bit fields are formed. res0[15:0] represents the least significant 16 bits of the PRNG output and this syntax applies to the other three fields. These fields are respectively substituted into the cf field (cf means a comparison field), and the comparison fields cf0,..., cf3 are each 16 bits long.

3 is compared with the probability value from the source operand src1[15:0] using the compare logic 40 to compare the four weighted random bits wpb[0],..., Wpb. [3] is derived.

The above syntax means that the value “1” is assigned to the bit wpb[0] when cf0<src1. Here, cf0 is a 16-bit unsigned random value and can have a value in the range {0,...,65535}. src1 is a 32-bit wide operand, but only the least significant 17 bits are used for rmask. Values in the range {0-65536} are allowed for src1, so the "unmasked" probability is src1/65536.

Note that for the allowed src1 range {0,...,65536}, the 17th bit of src1 (src1[16]) is set only for the value 65536. Since the maximum value of cf0 is 65535 and 65535<65536, wpb[0] automatically becomes “1” when src1[16]==1.

Then, the 4 bits wpb[0,..., 3] are used respectively to unmask the four 16-bit values in src0[63:0] respectively. Therefore, it becomes as follows.

16'b0 represents a 16-bit wide sequence having the value "0", 1'b0 means the bit with the value "0" and 1'b1 means the bit with the value "1".

For example, when wpb[1]==1, aDst[31:16]=aSrc0[31:16], and otherwise, aDst[31:16]=0.

In one embodiment, the probability value aSrc1 is used to select the probability that a value will be randomly “unmasked” (ie unmasked).

If the operand aSrc1=65536, then all values are always unmasked.

For the operand aSrc1=49152, each value is independently masked with a probability of 0.25 and unmasked with a probability of 0.75.

For operand aSrc1=32768, each value is independently masked with a probability of 0.50 and unmasked with a probability of 0.50.

For the operand aSrc1=16384, each value is masked with a probability of 0.75 and unmasked with a probability of 0.25.

If the operand aSrc1=0, all values are always masked.

"Unmasking" an input means reproducing the value represented by the input at the output. The output format and precision are not necessarily the same as the input format.

By "masking" an input is meant producing a symbol at the output that represents the value zero, rather than the value represented by the input.

Note that unmask probability == 1-mask probability.

In the implementation of rmask described herein, the probability value aSrc1==65536*unmask probability. The corresponding mask probability can be derived from the mask probability: 65536*(1-mask probability).

Therefore, it will be appreciated that either the mask probability or the unmask probability is defined (the sum of the two is "1").

The rmask instruction has several different uses. For example, when implementing dropout, the rmask instruction can be used just before writing the neural output to memory. All neurons in the next stage of the neural network then take the masked activity value from the previous layer.

When implementing a drop connection, the rmask instruction can be used immediately after reading the active value from memory just before calculating the dot product. Instructions have other uses than dropouts and connections as they do not depend on what the source vector represents. Also, this instruction does not depend on the format of the 16-bit or 32-bit scalar value in the source vector. On output, each scalar value remains unchanged or all bits are masked to zero. Therefore, mask instructions can be used to mask four 16-bit integers, signed or unsigned, or vectors in various 16-bit floating point formats. The only requirement for the scalar value format is that the value represented by all bits to zero be "0".

In some numeric formats, there can be multiple symbols representing the value zero.

For example, in the IEEE half-precision floating point number format, either of the following symbols can be used to represent zero:
16'b0000000000000000 (positive zero)
16'b1000000000000000 (negative zero)

The term "random" as used herein can mean "true random" or "pseudorandom." The rmask instruction can use either a pseudo-random bit sequence generator or a true random bit sequence generator.

Pseudo-random numbers are generated by a "pseudo-random number generator" or "PRNG". PRNGs can be implemented as software or hardware. The true random number is generated by a “true random number generator” or “TRNG”. An example of TRNG is a "transition effect ring oscillator". The advantage of PRNGs over TRNGs is determinism (running the same program twice with the same starting conditions always gives the same result).

The advantage of TRNG over PRNG is that the output is truly random (the output of PRNG satisfies a finite set of arbitrarily chosen mathematical properties, but the state and output of PRNG is always from the current state). Predictable and therefore not truly random).

Although particular embodiments have been described, other applications and variations of the disclosed techniques may become apparent to those skilled in the art after a disclosure hearing. The scope of the present disclosure is not limited by the above-described embodiments, but only by the appended claims.

Claims (18)

An execution unit for executing a computer program including a sequence of instructions, the sequence including masking instructions, the execution unit configured to execute the masking instructions, the masking instructions comprising the execution unit. , Masking randomly selected positions in the source operand having n values, and of each of the positions in the result, the selected position is a masked value and the selected An execution unit for generating a result that leaves the original value at the corresponding position of the source operand for positions other than the corresponding position . A hardware pseudo-random number generator (HPRNG) configured to generate a randomized bit string, the sequence of random bit fields for randomly selecting a position to be masked derived from the randomized bit string. The execution unit according to claim 1, which is executed. Each randomized bit string output from the HPRNG includes m bits, and the execution unit divides the m bits into n bit fields and compares each bit field with a probability value to obtain the value. The execution unit according to claim 2, wherein the execution unit is configured to generate a weighted probability indicator for selectively masking. 4. The execution unit of claim 3, wherein the masking instruction includes a source field that identifies the source operand , an indication of a destination register for holding the result, and a probability field that defines the probability value. The execution unit according to any one of claims 1 to 4, comprising an input buffer for holding the source operand, and an output buffer for holding the result. The execution unit according to claim 1, wherein the value of the source operand represents a weight of a link in a neural network. The execution unit according to claim 1, wherein the value in the source operand represents an activation value that defines an output value of a node in a neural network. 8. An execution unit according to any one of claims 1 to 7, wherein the sequence of instructions comprises instructions for implementing a dot product calculation of the result and a further set of values. 8. An execution unit according to any of claims 1 to 7, wherein the sequence of instructions comprises instructions for writing the result to a memory location after the masking instruction has been executed. The execution unit according to any one of claims 1 to 9, wherein the source operand comprises four 16-bit values. The execution unit according to any one of claims 1 to 9, wherein the source operand comprises two 32-bit values. A method of executing a computer program including a sequence of instructions, the sequence including a masking instruction, the method randomly responsive to execution of the masking instruction to randomly locate positions in a source operand having n values. Selecting and masking, and of the respective positions in the result, the selected positions are masked values, and the positions other than the selected positions are in the corresponding positions of the source operand. A method that includes producing a result that is left at its original value . A weighted probability indicator for providing a randomized bit string output containing m bits, dividing the m bits into n bit fields and comparing each bit field with a probability value to selectively mask the position. 13. The method of claim 12, comprising generating. 14. The method according to claim 12 or 13, comprising generating a randomized bit string, wherein a sequence of random bit fields for randomly selecting the masked positions is derived from the randomized bit string. 14. The method of claim 13, wherein the masking instruction includes a source field that identifies the source operand , an indication of a destination register to hold the result, and a probability field that defines the probability value. 16. The method of any one of claims 12-15, wherein the sequence of instructions includes instructions for implementing a dot product calculation of the result and a further set of values. 16. The method of any of claims 12-15, wherein the sequence of instructions includes instructions for writing the result to a memory location after the masking instruction has been executed. A program executed by a computer including a sequence of computer readable instructions stored on a non-transmission medium, the sequence including a masking instruction, the masking instruction causing the computer to: in a source operand having n values . A position is randomly selected to be a mask target, and among the respective positions in the result, the selected position is set to a masked value, and the positions other than the selected position are set to the source operand. A program that is responsible for performing the generation of the result with the original value at the corresponding position .