JP2018109870A - Information processor, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve both maintenance of entire linkages between specific layers in a neural network and reduction of the number of parameters.SOLUTION: An information processor includes: a first calculation part which executes calculation of forward propagation of a multilayer neural network including first layers 101-107, second layers 111-117 for receiving output from the first layers 101-107 and third layers 121-127 for receiving output from the second layers; and a second calculation part which executes calculation of backward propagation of the multilayer neural network based on the calculation result of forward propagation. An output from each unit 101-107 of the first layer is input into all units 121-127 of the third layer via the units 111-117 of the second layer. The number of units of the second layer is greater than the number of edges between each unit 101-107 of the first layer and the second layer and the number of edges between each unit 121-127 of the third layer and the second layer.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a machine learning technique.

  A multilayer neural network (for example, a convolutional neural network described in the following document) is used for classification of image data and audio data. In a multilayer neural network used as a classifier, a plurality of units in one layer and a plurality of units in another adjacent layer are fully coupled. A layer in which such a total bond is built is called a total bond layer.

  In a multilayer neural network, parameters (specifically, edge weights) are prepared for each edge. Therefore, the number of parameters in all connected layers is larger than the number of parameters in other layers. May be occupied by the parameters of the total coupling layer. However, an increase in the number of parameters leads to an increase in memory consumption. In addition, when a plurality of computers perform parallel neural network calculations in parallel, the amount of communication data transferred between the computers increases as the number of parameters increases, and the time until the calculation is finally completed becomes longer.

JP, 2006-33806, A

  An object of the present invention is, in one aspect, to provide a technique for achieving both the maintenance of total coupling between specific layers in a neural network and the reduction of the number of parameters.

  An information processing apparatus according to one aspect includes a first layer, a second layer that receives an output from the first layer, and a third layer that receives an output from the second layer. A first calculation unit that executes forward propagation calculation; and a second calculation unit that executes reverse propagation calculation of the multilayer neural network based on the result of forward propagation calculation. Then, the output from each unit of the first layer is input to all units of the third layer via the unit of the second layer. Also, the number of units in the second layer depends on the number of edges between each unit in the first layer and the second layer and the number of edges between each unit in the third layer and the second layer. More than the number.

  In one aspect, it is possible to simultaneously maintain the total coupling between specific layers in the neural network and reduce the number of parameters.

FIG. 1 is a diagram illustrating an example of total coupling. FIG. 2 is a diagram illustrating a correspondence relationship between the edges of the all coupling layers and the weight matrix. FIG. 3 is a diagram illustrating an example of a Latin square full coupling layer. FIG. 4 is a diagram illustrating an overview of the parallel computing system according to the first embodiment. FIG. 5 is a functional block diagram of the information processing apparatus. FIG. 6 is a diagram for explaining forward propagation and reverse propagation. FIG. 7 is a diagram illustrating a processing flow of processing for forward propagation. FIG. 8 is a diagram for explaining processing for forward propagation. FIG. 9 is a diagram showing a processing flow of the processing of the Latin square full coupling layer for forward propagation. FIG. 10 is a diagram for explaining a matrix product executed for a normal fully connected layer. FIG. 11 is a diagram for explaining a matrix product executed for a Latin square fully connected layer. FIG. 12 is a diagram illustrating a processing flow of processing for back propagation. FIG. 13 is a diagram for explaining processing for back propagation. FIG. 14 is a diagram illustrating a processing flow of processing of the Latin square full coupling layer for back propagation. FIG. 15 is a diagram for explaining the mask matrix. FIG. 16 is a diagram for explaining the mask matrix. FIG. 17 is a diagram for explaining the mask matrix. FIG. 18 is a diagram for explaining the mask matrix. FIG. 19 is a diagram for explaining the mask matrix. FIG. 20 is a diagram for explaining the mask matrix. FIG. 21 is a diagram for explaining the mask matrix. FIG. 22 is a diagram for explaining the mask matrix. FIG. 23 is a diagram illustrating a processing flow of processing for back propagation. FIG. 24 is a diagram illustrating the relationship between the number of information processing apparatuses and the time required for parallel calculation. FIG. 25 is a diagram for explaining the reduction in the number of edges. FIG. 26 is a diagram for explaining the reduction of the number of edges. FIG. 27 is a diagram illustrating a correspondence relationship between the Latin square full coupling layer and the mask matrix. FIG. 28 is a diagram for explaining control of the number of units on the input side and the number of units on the output side. FIG. 29 is a diagram illustrating an overview of a system according to the third embodiment. FIG. 30 is a diagram for explaining a Latin square fat tree and a finite projection plane. FIG. 31 is a diagram for explaining a Latin square fat tree and a finite projection plane. FIG. 32 is a diagram for explaining a Latin square fat tree and a finite projection plane. FIG. 33 is a diagram for explaining a Latin square fat tree and a finite projection plane. FIG. 34 is a functional block diagram of a computer.

[Embodiment 1]
FIG. 1 shows an example of total coupling. In FIG. 1, a circle represents a unit, and a line segment between units represents an edge. In FIG. 1, each unit in the (l + 1) th layer is connected to all units in the lth layer. Since the number of units in each layer is 7, the number of edges is 49 (= 7 * 7). In machine learning, parameters (that is, weights) of each edge are updated, and weights between layers are expressed as a matrix. For example, in the example of FIG. 1, the weight matrix is a 7 × 7 matrix. The input to the (l + 1) th layer in FIG. 1 is calculated using a weight matrix that is a dense matrix and the output of the lth layer.

  However, the weight matrix after machine learning is a sparse matrix for the fully connected layer. For example, among the 16 edges between the l-th layer having 4 units and the (l + 1) -th layer having 4 units shown in FIG. 2A, the weight value after machine learning is zero. It is assumed that an edge that is not (or almost zero) is a thick line edge. In this case, the weight matrix corresponding to the weight between layers is a sparse matrix as shown in FIG. In FIG. 2B, the position in the row direction corresponds to the position of the (l + 1) th layer unit, and the position in the column direction corresponds to the position of the lth layer unit. An element corresponding to an edge whose weight value is not zero (or almost zero) is hatched. As described above, it is considered that the edges of all the connection layers in the multilayer neural network (hereinafter referred to as DNN (Deep Neural Network)) include potentially useless edges.

  Therefore, in the present embodiment, a Latin Square Fully-Connected Layer (LSFCL) is introduced into DNN. FIG. 3 shows an example of a Latin square full coupling layer. In the Latin square fully connected layer, the edge is set so that the output from each unit in the l-th layer is input to all the units in the (l + 2) -th layer. For example, the output from the unit l01 in the l-th layer is input to the unit l11, the unit l12, and the unit l13 in the (l + 1) -th layer. Furthermore, the output from the unit l11 is input to the unit l21, the unit l22, and the unit l23 in the (l + 2) th layer. The output from the unit l12 is input to the unit l21, the unit l24, and the unit l25. The output from the unit l13 is input to the unit l21, the unit l26, and the unit l27. As a result, the output from the unit 101 is input to all units in the (l + 2) layer. The same applies to the other units in the l-th layer.

  The number of edges of the Latin square full coupling layer shown in FIG. 3 is 42 (= (3 * 7) * 2). Since the number of edges of the Latin square full coupling layer is smaller than the number of edges of the total coupling layer, the number of weight values held in a storage device such as a memory as a parameter can be reduced.

  Please refer to the appendix for matters related to Latin squares.

  Below, the specific content of this Embodiment is demonstrated. FIG. 4 shows an overview of the parallel computing system 1000 of the present embodiment. The parallel computing system 1000 is a system that executes DNN calculations in parallel, and includes information processing apparatuses 1a to 1c that are, for example, physical servers or personal computers. The information processing apparatuses 1a to 1c are connected to a network 5 that is a LAN (Local Area Network), for example. Each information processing apparatus processes different data, transmits a processing result (for example, data used for parameter update) to another information processing apparatus, and receives a processing result from the other information processing apparatus. . Each information processing apparatus updates DNN parameters based on the collected processing results. In FIG. 4, the number of information processing apparatuses is three, but the number is not limited.

  FIG. 5 shows a functional block diagram of the information processing apparatus 1a. The information processing apparatus 1a includes a first calculation unit 101, a second calculation unit 103, a communication unit 105, a data storage unit 111, a target output storage unit 113, and a work data storage unit 115. The functional block diagrams of the information processing apparatus 1b and the information processing apparatus 1c are the same as the functional block diagram of the information processing apparatus 1a. The first calculation unit 101, the second calculation unit 103, and the communication unit 105 are realized by a CPU (Central Processing Unit) 2503 in FIG. 34 executing a program. The data storage unit 111, the target output storage unit 113, and the work data storage unit 115 are provided in the memory 2501 or the HDD 2505 (Hard Disk Drive) in FIG.

  The data storage unit 111 stores data (for example, image data, audio data, etc.) that are targets of machine learning and classification. The target output storage unit 113 stores a target output (also called a label). The first calculation unit 101 performs forward propagation calculation on the data stored in the data storage unit 111 and stores the calculation result in the work data storage unit 115. The second calculation unit 103 performs back propagation calculation based on the data stored in the target output storage unit 113 and the data stored in the work data storage unit 115, and the processing result (for example, parameter error) is obtained. It is stored in the work data storage unit 115. The communication unit 105 transmits the data stored in the work data storage unit 115 to the information processing devices 1b and 1c, and stores the data received from the information processing devices 1b and 1c in the work data storage unit 115. The second calculation unit 103 updates parameters based on data stored in the work data storage unit 115.

  As shown in FIG. 6, forward propagation is a process for calculating an output from an input of DNN. Back propagation is a process of calculating an error of a parameter from an error between a calculated output and a target output. Based on the calculated parameter error, the DNN parameter is updated. As shown in FIG. 6, the back propagation calculation proceeds in the opposite direction to the forward propagation calculation.

  Next, processing executed by each information processing apparatus will be described in detail with reference to FIGS. In the following, processing executed by the information processing apparatus 1a will be described as an example, but processing executed by an information processing apparatus other than the information processing apparatus 1a is the same as processing executed by the information processing apparatus 1a.

  First, processing for forward propagation will be described with reference to FIGS.

  The first calculation unit 101 of the information processing apparatus 1a reads out data to be processed from the data storage unit 111 (FIG. 7: step S1).

  The first calculation unit 101 sets 0 to i, which is a variable representing the layer number (step S3).

  The first calculation unit 101 determines whether or not the i-th layer is a Latin square fully connected layer (step S5). Here, when the i-th layer corresponds to the (l + 1) -th layer (that is, the intermediate layer) in FIG. 3, it is determined that the i-th layer is a Latin square full coupling layer.

  When the i-th layer is not a Latin square all coupled layer (step S5: No route), the first calculation unit 101 executes the i-th layer operation for forward propagation (step S7), and the processing result is stored in the work data storage unit. 115. Then, the process proceeds to step S11.

  The calculation for forward propagation will be described with reference to FIG. The memory 2501 stores data used for processing of each layer and processing results of each layer. The arrow represents a pointer. The data read in step S1 is stored in the area “Data-0”.

  In the layer 0, processing is executed using the data stored in the area “Data-0” as input data and the data stored in the area “Weight-0” as weights, and the processing result is output as output data in the area “Data-”. 1 ".

  The above calculation is expressed by the following formula when the batch size is 1.

  u represents an input, w represents a weight, and z represents an output. When the batch size is m (m is a natural number), the above calculation is expressed by the following equation.

  In the layer 1, processing is executed using the data stored in the area “Data-1” as input data and the data stored in the area “Weight-1” as weights, and the processing result is output as output data in the area “Data-”. 2 ".

  In the layer 2, processing is executed using the data stored in the area “Data-2” as input data and the data stored in the area “Weight-2” as weights, and the processing result is output as output data in the area “Data−”. 3 ".

  In the layer 3, processing is executed using the data stored in the area “Data-3” as input data and the data stored in the area “Weight-3” as weights, and the processing result is output as output data in the area “Data-”. 4 ".

  In this way, the input to the DNN is stored at the start point of the chain pointer, and the output from the DNN is stored at the end point of the chain pointer.

  Returning to the description of FIG. 7, when the i-th layer is the Latin square full connection layer (step S5: Yes route), the first calculation unit 101 performs the operation of the Latin square full connection layer for forward propagation (step S9). ). The process of step S9 will be described with reference to FIGS.

  The first calculation unit 101 reads data as input data from the data storage area (that is, the area “Data-i”) in the work data storage unit 115 (FIG. 9: step S21).

  The first calculation unit 101 calculates the first weight matrix and the second weight matrix from the weight storage area (that is, the area “Weight-i” and the area “Weight- (i + 1)”) in the work data storage unit 115. Is read (step S23).

  The first calculation unit 101 calculates a matrix product of the matrix of input data read in step S21 and the first weight matrix read in step S23 (step S25). The result of the calculation is stored in a storage area (for example, area “Data− (i + 1)”) in the work data storage unit 115.

  The first calculation unit 101 calculates the matrix product of the matrix product result in step S25 and the second weight matrix read in step S23 (step S27).

  The matrix product executed in the DNN calculation will be described with reference to FIGS. With reference to FIG. 10, a matrix product executed for a normal fully connected layer will be described. In FIG. 10, the output from the l-th layer is input to the (l + 1) -th layer, and each output is multiplied by a weight. Specifically, if the output from the l-th layer is input X, the weight matrix is W, and the input to the (l + 1) -th layer is output Z, the output is calculated as X * W → Z.

With reference to FIG. 11, the matrix product executed for the Latin square fully connected layer of the present embodiment will be described. In FIG. 11, each output from the l-th layer is multiplied by a weight and input to the (l + 1) -th layer. Next, each output from the (l + 1) th layer is multiplied by a weight and input to the (l + 2) th layer. That is, the matrix product is executed twice. The weight matrix used for the first matrix product is a first weight matrix (M ₁ ), and the weight matrix used for the second matrix product is a second weight matrix (M ₂ ). Then, in the first matrix product, Y is calculated as X * M ₁ → Y, and in the second matrix product, Z is calculated as Y * M ₂ → Z. That is, as a whole, Z is calculated as X * M ₁ * M ₂ → Z.

  In the present embodiment, since total coupling is realized using three layers, Z is calculated using two weight matrices.

  Returning to the description of FIG. 9, the first calculation unit 101 writes the result of the matrix product calculated in step S27 as output data in the data storage area (that is, the area “Data− (i + 2)”) (step S29). .

  Since the first calculation unit 101 has executed processing for two layers, i is incremented by 1 (step S30). Processing then returns to the caller.

  Returning to the description of FIG. 7, the first calculation unit 101 increments i by 1 (step S11). The first calculation unit 101 determines whether i <N is satisfied (step S13). N is a natural number of 3 or more, and is the number of layers of DNN.

  When i <N is satisfied (step S13: Yes route), the process returns to step S5. On the other hand, when i <N is not satisfied (step S13: No route), the first calculation unit 101 outputs the operation result of DNN (step S15). Then, the process ends. For example, during machine learning, the first calculation unit 101 stores the DNN calculation result in the work data storage unit 115. For example, when data is classified, the first calculation unit 101 displays the DNN calculation result on the display device and stores the DNN calculation result in the work data storage unit 115.

  By executing the processing as described above, the forward propagation calculation can be appropriately executed for the DNN including the Latin square full coupling layer.

  Next, processing for back propagation will be described with reference to FIGS.

  The second calculation unit 103 of the information processing device 1a calculates error data based on the difference between the DNN calculation result stored in the work data storage unit 115 and the target output stored in the target output storage unit 113 ( FIG. 12: Step S31), the calculated error data is stored in the error storage area in the work data storage unit 115.

  The second calculation unit 103 sets (N−1) to i, which is a variable representing the layer number (step S33).

  The second calculation unit 103 determines whether or not the i-th layer is a Latin square fully connected layer (step S35). Here, when the i-th layer corresponds to the (l + 1) -th layer (that is, the intermediate layer) in FIG. 3, it is determined that the i-th layer is a Latin square full coupling layer.

  When the i-th layer is not a Latin square all-coupled layer (step S35: No route), the second calculation unit 103 executes the i-th layer operation for back propagation (step S37), and the processing result is stored in the work data storage unit. 115. Then, the process proceeds to step S41.

  The calculation for back propagation will be described with reference to FIG. The memory 2501 stores data used for processing of each layer and processing results of each layer. The arrow represents a pointer.

  In the layer 2, processing is executed with the data stored in the region “Diff-2” as an input error and the data stored in the region “Weight-2” as a weight, and the processing result is output as an output error in the region “Diff−”. 1 ". Further, the process is executed using the data stored in the area “Diff-2” as the input error and the data stored in the area “Data-2” as the input data, and the processing result is used as the weight error in the area “DeltaW-2”. Is stored.

  The above calculation is expressed by the following formula when the batch size is 1.

  dz represents an input error, w represents a weight, du represents an output error, u represents an input, and dw represents a weight error. When the batch size is m (m is a natural number), the above calculation is expressed by the following equation.

  In the layer 1, processing is executed using the data stored in the region “Diff−1” as an input error and the data stored in the region “Weight-1” as a weight, and the processing result is output as an output error in the region “Diff−”. 0 ". Further, the process is executed using the data stored in the area “Diff-1” as an input error and the data stored in the area “Data-1” as input data, and the processing result is regarded as a weight error in the area “DeltaW-1”. Is stored.

  In the layer 0, the process is executed using the data stored in the region “Diff-0” as an input error and the data stored in the region “Weight-0” as a weight, and a final error is calculated. In addition, the process is executed using the data stored in the area “Diff-0” as the input error and the data stored in the area “Data-0” as the input data, and the processing result is regarded as a weight error in the area “DeltaW-0”. Is stored.

  The weight error matrix (hereinafter referred to as weight matrix error) calculated in step S37 is used for updating the weight matrix in each information processing apparatus.

  Returning to the description of FIG. 12, when the i-th layer is a Latin square full coupling layer (step S <b> 35: Yes route), the second calculation unit 103 performs a back propagation operation on the Latin square full coupling layer (step S <b> 39). ). The process of step S39 will be described with reference to FIGS.

  First, the second calculation unit 103 reads data as input error data from the error storage area (that is, the area “Diff-i”) in the work data storage unit 115 (FIG. 14: step S61).

The second calculation unit 103 calculates the second weight matrix and the first weight from the weight storage area (that is, the area “Weight-i” and the area “Weight- (i−1)”) in the work data storage unit 115. The matrix is read (step S63). In the following, the second weight matrix is M ₂ and the first weight matrix is M ₁ .

  The second calculator 103 calculates a matrix product of the matrix of the input error data read in step S61 and the second weight matrix read in step S63 (step S65). The calculation result is stored in a storage area (for example, the area “Diff− (i−1)”) in the work data storage unit 115.

  The second calculation unit 103 calculates the matrix product of the matrix product result in step S65 and the first weight matrix read in step S63 (step S67).

Thus, in steps S65 and S67, calculation in the opposite direction to S25 and S27 is performed. For example, if the input error data matrix is ΔZ, an intermediate error matrix ΔY is calculated as ΔZ * M ₂ → ΔY, and further, an output error data matrix is calculated as ΔY * M ₁ → ΔX.

  The second calculation unit 103 writes the result of the matrix product calculated in step S67 as output data in the data storage area (that is, the area “Diff− (i−2)”) in the work data storage unit 115 ( Step S69).

  The second calculation unit 103 reads data as input data from the data storage area (that is, the area “Data-i” and the area “Data- (i−1)”) (step S71).

Based on the matrix product of the input data matrix read in step S71 and the input error data matrix read in step S61, the second calculator 103 calculates the error of the second weight matrix and the first An error of the weight matrix is calculated (step S73). Specifically, if the input data read from the area “Data-i” in step S71 is Y and the matrix product result calculated in step S65 is ΔZ, Y * ΔZ → ΔM ₂ A weight matrix error ΔM ₂ is calculated. Further, the input data read from the area "Data- (i-1)" is X in step S71, when the result of the calculated matrix product in step S67 and [Delta] Y, the as X * ΔY → ΔM _{1 1} An error ΔM ₁ of the weight matrix is calculated.

  In the present embodiment, since total coupling is realized using three layers, errors of two weight matrices are calculated respectively.

  The second calculation unit 103 performs mask processing using the mask matrix on the error of the second weight matrix and the error of the first weight matrix calculated in step S73 (step S75), The error of the weight matrix and the error of the first weight matrix are updated. The mask matrix will be described later.

  The above calculation is expressed by the following formula when the batch size is 1.

  dw represents a weight error. When the batch size is m (m is a natural number), the above calculation is expressed by the following equation.

  The second calculation unit 103 converts the error of the second weight matrix and the error of the first weight matrix, for which mask processing has been performed in step S75, into error storage areas (for example, the area “Delta_Wi” and the area "Delta_W- (i-1)") (step S77).

  Since the second calculation unit 103 has executed processing for two layers, i is decremented by 1 (step S79). Processing returns to the caller.

  The mask matrix will be described with reference to FIGS.

First, consider generating a mask matrix for a Latin square fat tree as shown in FIG. The Latin square fat tree shown in FIG. 15 corresponds to half of the Latin square full coupling layer shown in FIG. The Latin square fat tree contains (n ² + n + 1) nodes, each node having (n + 1) edges. The Latin square fat tree shown in FIG. 15 corresponds to the Latin square fat tree in the case of n = 2.

  The Latin square fat tree shown in FIG. 15 includes two layers including seven nodes, and there may be 49 (= 7 * 7) edges. Accordingly, a mask matrix having 49 elements as shown in FIG. 16A can be considered. However, since there are actually only 21 edges, 21 elements among the 49 elements are specified. Then, by setting elements other than the specified element to 0 and using them as a mask matrix, the matrix subjected to mask processing is converted into a sparse matrix.

As shown in FIG. 16B, the mask matrix is divided into a lower right n ² * n ² region and other portions. First, an element is specified from other parts. First, (n + 1) elements are specified as a mask for the first row.

  Next, as shown in FIG. 17A, n elements are specified as a mask for the second line, and n elements are specified as a mask for the third line.

The above operation is also executed for the columns. That is, as shown in FIG. 17B, (n + 1) elements are specified as a mask for the first column. However, since one element has already been specified, two new elements are specified as masks here. Then, as shown in FIG. 18, n elements are specified as masks for the second column, and n elements are specified as masks for the third column. By the operation so far, the mask specification is completed for the region other than the region of n ² * n ² .

Next, a mask is specified from the region of n ² * n ² . First, as shown in FIG. 19A, the n ² * n ² region is divided into n * n blocks. Then, as shown in FIG. 19B, a two-dimensional ID (x, y) is assigned to each block. (0,0) is assigned to the upper left block, (0,1) is assigned to the lower left block, (1,0) is assigned to the upper right block, and lower right (1, 1) is assigned to this block.

  Then, for the i-th (i is 0 ≦ i ≦ (n−1)) row in each block of n * n, the ((i + (x * y)) mod 2) column elements are specified as a mask. For example, for the upper left block whose ID is (0, 0), ((0+ (0 * 0)) mod 2) = 0 for the 0th row (here, the upper row of the two rows). Therefore, the element in the 0th column (here, the left column of the two columns) is specified as a mask. Since ((1+ (0 * 0)) mod 2) = 1 holds for the first row, the element in the first column (here, the right column of the two columns) is specified as a mask. . The mask matrix shown in FIG. 20 is completed by the above processing. The generated mask matrix is applied to both the first weight matrix and the second weight matrix.

Next, a method for generating a mask matrix having a width of (n ² + n + 1) when n = 3 will be described with reference to FIG.

In FIG. 21, the mask matrix 2101 is divided into an n ² * n ² area and other areas, and diagonal elements are initially specified as masks in the n ² * n ² area. The n ² * n ² region is divided into n * n blocks, and the diagonal elements in each block are right-shifted according to Galois field (here, GF (3)) 2103. Then, the mask matrix 2102 is completed.

  A Galois field is a set with a finite number of elements and is also called a finite field. GF (n) is generated by calculating mod n for the calculation result. A Galois field corresponds to a finite projective plane of a Latin square fat tree. The Galois field is used for communication error correction and AES (Advanced Encryption Standard). FIG. 22 shows an example of GF (5). 22A shows a Galois field generated by calculating mod 5 for the sum of element numbers, and FIG. 22B shows mod 5 for the product of element numbers. The Galois field generated by computing is shown.

  Returning to the description of FIG. 12, the second calculation unit 103 decrements i by 1 (step S41). The second calculation unit 103 determines whether i ≧ 0 is satisfied (step S43).

  If i ≧ 0 holds (step S43: Yes route), the process returns to step S35. On the other hand, if i ≧ 0 does not hold (step S43: No route), the process proceeds to step S45 in FIG.

  Shifting to the description of FIG. 23, the communication unit 105 receives the error of the weight matrix calculated in the other information processing apparatuses (that is, the information processing apparatuses 1 b and 1 c) from the other information processing apparatuses, and the information processing apparatus The error of the weight matrix calculated by the second calculation unit 103 of 1a is transmitted to another information processing apparatus (step S45).

  The second calculation unit 103 sets (N−1) to i, which is a variable representing the layer number (step S47).

  The second calculation unit 103 updates the weight matrix for the i-th layer using the error matrix of the weight matrix (step S49). For example, if the error of the weight matrix is ΔW and the weight matrix is W, the weight matrix is updated as W−ΔW → W. However, the weighting matrix may be updated by further using a learning rate, a bias, or the like.

  The second calculation unit 103 decrements i by 1 (step S51). The second calculation unit 103 determines whether i ≧ 0 is satisfied (step S53). If i ≧ 0 holds (step S53: Yes route), the process returns to step S49. On the other hand, when i ≧ 0 is not satisfied (step S53: No route), the process ends.

  As described above, according to the method of the present embodiment, the number of edges of the part that is fully combined in the DNN is reduced, so that the number of weights that are parameters is reduced. As a result, the consumption of the capacity of the storage device such as the memory 2501 can be suppressed, and the calculation amount can be reduced. Therefore, it is also effective for operation using a computer with scarce physical resources such as a processor and a memory. Furthermore, since the amount of data exchanged between information processing devices in communication is reduced, the time required for DNN parallel computation can be reduced.

  FIG. 24 shows the relationship between the number of information processing apparatuses and the time required for parallel calculation. In FIG. 24, the vertical axis represents time, and the horizontal axis represents the number of information processing apparatuses. As shown in FIG. 24, if the number of information processing apparatuses is increased, the time required for calculation is shortened. However, even if the number of information processing apparatuses is increased, the time required for communication is not shortened. Therefore, the time required for communication becomes a problem as the number of information processing apparatuses increases. However, according to the method of the present embodiment, the time required for communication can be shortened, so that the effect of improving the performance of parallel computing is great particularly when the number of information processing apparatuses is large.

Note that the Latin square full coupling layer cannot be constructed with an arbitrary number of units. In principle, each unit has (n + 1) edges, and the number of nodes at that time is (n ² + n + 1). When the number of units is (n ² + n + 1), the total number of edges is 2 (n ² + n + 1) (n + 1). On the other hand, when the number of units of the normal total coupling layer is (n ² + n + 1), the total number of edges is (n ² + n + 1) ² .

  Based on this premise, the reduction in the number of edges will be described with reference to FIGS. 25 and 26. FIG. 25 is a diagram illustrating the relationship between the number of edges and the number of units in the case of a Latin square full coupling layer and a normal full coupling layer. In FIG. 25, the vertical axis represents the number of edges, and the horizontal axis represents the number of units. For example, when the number of units is about 1000, the number of edges of the Latin square full bonding layer is about 100,000, and the number of edges of a normal full bonding layer is about 1,000,000.

  FIG. 26 is a diagram showing the ratio between the number of edges of the Latin square full coupling layer and the number of edges of the normal full coupling layer. In FIG. 26, the vertical axis represents the ratio, and the horizontal axis represents the number of units. If the ratio is less than 1, the number of edges in the Latin square full coupling layer is less than the number of edges in the normal full coupling layer. As described above, except for the case where the number of units is extremely small, the number of edges of the Latin square full coupling layer is smaller than the number of edges of the normal total coupling layer.

[Embodiment 2]
As shown in FIG. 27, the Latin square full coupling layer in the first embodiment has a shape in which two Latin square fat trees are connected. The mask matrix 2701 corresponds to the left edge group, and the mask matrix 2702 corresponds to the right edge group. Filled elements correspond to edges. The output from the unit u27 in the left layer is output to all the units in the right layer via the unit in the middle layer. In FIG. 27, the number of units in the left layer, the number of units in the middle layer, and the number of units in the right layer are equal.

However, the number of units in the left layer and the number of units in the right layer are not necessarily the same as the number of units in the intermediate layer. For example, as shown in FIG. 28, a unit included in a portion surrounded by a broken line among units on the left layer, a unit included in a middle layer, and a unit surrounded by a broken line among units in a right layer. A Latin square full connective layer may be constructed with units. In this case, the number of rows in each mask matrix is reduced according to the number of units. The number of units in the middle layer is (n ² + n + 1), the number of units included in the portion surrounded by the broken line among the units in the right layer, and the portion surrounded by the broken line in the units in the left layer The smallest n is searched so as to be equal to or greater than the number of units to be processed.

  As a result, the Latin square full coupling layer of the present embodiment can be applied to any total coupling layer.

[Embodiment 3]
The parallel computing system 1000 according to the first embodiment includes a plurality of information processing apparatuses, and the plurality of information processing apparatuses are connected via a network. However, the form of the system is not limited to such a form. For example, as shown in FIG. 29, a system in which processors 10a to 10c are connected via a bus or the like in the information processing apparatus 1 and the processors 10a to 10c execute parallel computation may be used.

  Although one embodiment of the present invention has been described above, the present invention is not limited to this. For example, the functional block configuration of the information processing apparatus 1a described above may not match the actual program module configuration.

  Further, the configurations of the information processing apparatuses 1a to 1c and the information processing apparatus 1 described above are merely examples, and the configuration as described above is not necessarily required. Further, in the processing flow, the processing order can be changed if the processing result does not change. Further, it may be executed in parallel.

[Appendix]
This appendix describes the Latin square fat tree and the finite projective plane.

  The structure of the finite projective plane is replaced with a topology structure. The topology structure shown in FIG. 30A is replaced with the structure of the finite projection plane shown in FIG. In FIG. 30A, a hatched rectangle represents a spine switch, an unhatched rectangle represents a leaf switch, and a circle represents a server. In FIG. 30B, a straight line represents a spine switch, and a point represents a leaf switch.

  The topology structure shown in FIG. 31A is a Latin square fat tree topology structure in which the number of spine switches is 7 and the number of leaf switches is 7, and the finite projection plane shown in FIG. Corresponds to the structure of In FIG. 31A, the topology structure of the portion surrounded by the thick line is the same as the topology structure of FIG. Further, the structure of the portion surrounded by the thick line in FIG. 31B corresponds to the topology structure of the portion surrounded by the thick line in FIG.

  The finite projective plane corresponds to a plane obtained by adding several infinity points to an ordinary plane and eliminating “two parallel straight lines”. FIG. 32 shows the structure of the finite projection plane when the order (hereinafter referred to as n) is 2 and the number of ports is 6 (= 2 (n + 1)). In FIG. 32, 3 (= n + 1) leaf switches other than the 4 (= n * n) leaf switches on the hatched rectangle correspond to infinity points.

In the finite projective plane, there are (n ² + n + 1) points. The number of straight lines is (n ² + n + 1). Any two straight lines intersect at one point, and there is only one straight line connecting any two points. However, there is a restriction that n is a prime number.

  The structure shown in FIG. 32 can be converted into the structure shown in FIG. In FIG. 33, the hatched lattice portion corresponds to the hatched portion in FIG. Parallel lines in the grid portion are transformed to meet at additional points. That is, conversion is performed so that straight lines having the same inclination intersect.

  This completes the appendix.

  The information processing apparatuses 1a to 1c described above are computer apparatuses, and as shown in FIG. 34, a display control unit 2507 and a removable disk 2511 connected to a memory 2501, a CPU 2503, an HDD 2505, and a display apparatus 2509. Drive device 2513, input device 2515, and communication control unit 2517 for connecting to a network are connected by a bus 2519. An operating system (OS) and an application program for executing the processing in this embodiment are stored in the HDD 2505, and are read from the HDD 2505 to the memory 2501 when executed by the CPU 2503. The CPU 2503 controls the display control unit 2507, the communication control unit 2517, and the drive device 2513 according to the processing content of the application program, and performs a predetermined operation. Further, data in the middle of processing is mainly stored in the memory 2501, but may be stored in the HDD 2505. In the embodiment of the present invention, an application program for performing the above-described processing is stored in a computer-readable removable disk 2511 and distributed, and installed in the HDD 2505 from the drive device 2513. In some cases, the HDD 2505 may be installed via a network such as the Internet and the communication control unit 2517. Such a computer apparatus realizes various functions as described above by organically cooperating hardware such as the CPU 2503 and the memory 2501 described above and programs such as the OS and application programs. .

  The embodiment of the present invention described above is summarized as follows.

  The information processing apparatus (for example, information processing apparatus 1a) according to the first aspect of the present embodiment includes (A) a first layer, a second layer that receives an output from the first layer, and a second layer A first calculation unit (for example, the first calculation unit 101) that executes a forward propagation calculation of a multilayer neural network including a third layer that receives an output from the first layer; and (B) a result of the forward propagation calculation. And a second calculation unit (for example, the second calculation unit 103) that performs back propagation calculation of the multilayer neural network. Then, the output from each unit of the first layer is input to all units of the third layer via the unit of the second layer. Also, the number of units in the second layer depends on the number of edges between each unit in the first layer and the second layer and the number of edges between each unit in the third layer and the second layer. More than the number.

  Since the number of edges is reduced while the first layer and the third layer are fully coupled, the number of parameters (for example, weights corresponding to the edges) can be reduced. Also, the amount of calculation can be reduced by reducing the number of edges.

  In addition, the second calculation unit (b1) performs the first propagation corresponding to the edge between the first layer and the second layer by the back propagation calculation using the result of the forward propagation calculation and the target output. Calculating a weight error matrix and a second weight error matrix corresponding to an edge between the second layer and the third layer; and (b2) a first weight error matrix and a The first weight error matrix is converted to a first sparse matrix and the second weight error matrix is converted to a second sparse matrix by masking the error matrix of 2 weights (b3 ) The first weight matrix may be updated based on the first sparse matrix, and (b4) the second weight matrix may be updated based on the second sparse matrix.

  Even in the multilayer neural network as described above, the weight can be appropriately updated. In addition, the calculation can be simplified by using a sparse matrix.

  In the mask process, a Kronecker product of the first weight error matrix and the mask matrix and a Kronecker product of the second weight error matrix and the mask matrix may be executed.

  Further, the mask matrix may be a matrix in which elements corresponding to existing edges are set to a predetermined value and elements corresponding to non-existing edges are set to zero.

  The first calculation unit calculates (a1) a first matrix product of the output from the first layer and a first weight matrix, and (a2) a result of the first matrix product, The output from the third layer may be calculated based on a second matrix product with a weight of 2 matrix. Even in the multilayer neural network as described above, it is possible to appropriately classify input data.

  In addition, the information processing apparatus transmits the result of (C) the back propagation calculation to another information processing apparatus in the parallel computing system including the information processing apparatus, and the back propagation executed in the other information processing apparatus. You may further have a communication part (for example, communication part 105) which receives the result of calculation of from other information processing apparatuses. Since the number of parameters is reduced, the amount of communication data is reduced, and the time required for communication can be shortened.

The number of edges between each unit of the first layer and the second layer and the number of edges between each unit of the third layer and the second layer are (n (n is a natural number) +1) ), And the number of units in the second layer may be (n ² + n + 1).

  The information processing method according to the second aspect of the present embodiment includes (D) a first layer, a second layer that receives an output from the first layer, and an output that receives an output from the second layer. And (E) a process of executing a back propagation calculation of the multilayer neural network based on a result of the forward propagation calculation. Then, the output from each unit of the first layer is input to all units of the third layer via the unit of the second layer. Also, the number of units in the second layer depends on the number of edges between each unit in the first layer and the second layer and the number of edges between each unit in the third layer and the second layer. More than the number.

  A program for causing the processor to perform the processing according to the above method can be created, and the program can be a computer-readable storage medium such as a flexible disk, a CD-ROM, a magneto-optical disk, a semiconductor memory, a hard disk, or the like. It is stored in a storage device. The intermediate processing result is temporarily stored in a storage device such as a main memory.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Appendix 1)
Perform forward propagation computation of a multi-layer neural network including a first layer, a second layer that accepts output from the first layer, and a third layer that accepts output from the second layer A first calculation unit;
A second calculation unit that performs a back propagation calculation of the multilayer neural network based on the result of the forward propagation calculation;
Have
The output from each unit of the first layer is input to all units of the third layer via the unit of the second layer,
The number of units of the second layer includes the number of edges between each unit of the first layer and the second layer and between each unit of the third layer and the second layer. More than the number of edges,
Information processing device.

(Appendix 2)
The second calculator is
A matrix of first weight errors corresponding to an edge between the first layer and the second layer by the back propagation calculation using the result of the forward propagation calculation and a target output; Calculating a second weight error matrix corresponding to an edge between the second layer and the third layer;
The first weight error matrix is converted into a first sparse matrix by masking the first weight error matrix and the second weight error matrix and the second weight error To a second sparse matrix,
Updating the first weight matrix based on the first sparse matrix;
Updating the second weight matrix based on the second sparse matrix;
The information processing apparatus according to attachment 1.

(Appendix 3)
In the mask process,
Performing a Kronecker product of the first weight error matrix and a mask matrix and a Kronecker product of the second weight error matrix and the mask matrix;
The information processing apparatus according to attachment 2.

(Appendix 4)
The mask matrix is a matrix in which elements corresponding to existing edges are set to a predetermined value and elements corresponding to non-existing edges are set to zero.
The information processing apparatus according to attachment 3.

(Appendix 5)
The first calculator is
Calculating a first matrix product of the output from the first layer and the first weight matrix;
Calculating an output from the third layer based on a second matrix product of the first matrix product result and the second weight matrix;
The information processing apparatus according to any one of appendices 2 to 4.

(Appendix 6)
The result of the back propagation calculation is transmitted to another information processing apparatus in a parallel computing system including the information processing apparatus, and the result of the back propagation calculation executed in the other information processing apparatus is transmitted to the other information processing apparatus. The information processing apparatus according to any one of supplementary notes 1 to 5, further comprising: a communication unit that receives the information processing apparatus from the information processing apparatus.

(Appendix 7)
The number of edges between each unit of the first layer and the second layer, and the number of edges between each unit of the third layer and the second layer are (n (n is a natural number) ) +1),
The number of units of the second layer is (n ² + n + 1),
The information processing apparatus according to any one of appendices 1 to 6.

(Appendix 8)
Computer
Performing a forward propagation calculation of a multi-layer neural network including a first layer, a second layer receiving an output from the first layer, and a third layer receiving an output from the second layer ,
Performing back propagation calculations of the multilayer neural network based on the results of the forward propagation calculations;
Execute the process,
The output from each unit of the first layer is input to all units of the third layer via the unit of the second layer,
The number of units of the second layer includes the number of edges between each unit of the first layer and the second layer and between each unit of the third layer and the second layer. More than the number of edges,
Information processing method.

(Appendix 9)
On the computer,
Performing a forward propagation calculation of a multi-layer neural network including a first layer, a second layer receiving an output from the first layer, and a third layer receiving an output from the second layer ,
Performing back propagation calculations of the multilayer neural network based on the results of the forward propagation calculations;
Let the process run,
The output from each unit of the first layer is input to all units of the third layer via the unit of the second layer,
The number of units of the second layer includes the number of edges between each unit of the first layer and the second layer and between each unit of the third layer and the second layer. More than the number of edges,
program.

1, 1a, 1b, 1c Information processing device 5 Network 10a, 10b, 10c Processor 11 Bus 101 First calculation unit 103 Second calculation unit 105 Communication unit 111 Data storage unit 113 Target output storage unit 115 Work data storage unit

Claims (8)

Perform forward propagation computation of a multi-layer neural network including a first layer, a second layer that accepts output from the first layer, and a third layer that accepts output from the second layer A first calculation unit;
A second calculation unit that performs a back propagation calculation of the multilayer neural network based on the result of the forward propagation calculation;
Have
The output from each unit of the first layer is input to all units of the third layer via the unit of the second layer,
The number of units of the second layer includes the number of edges between each unit of the first layer and the second layer and between each unit of the third layer and the second layer. More than the number of edges,
Information processing device. The second calculator is
A matrix of first weight errors corresponding to an edge between the first layer and the second layer by the back propagation calculation using the result of the forward propagation calculation and a target output; Calculating a second weight error matrix corresponding to an edge between the second layer and the third layer;
The first weight error matrix is converted into a first sparse matrix by masking the first weight error matrix and the second weight error matrix and the second weight error To a second sparse matrix,
Updating the first weight matrix based on the first sparse matrix;
Updating the second weight matrix based on the second sparse matrix;
The information processing apparatus according to claim 1. In the mask process,
Performing a Kronecker product of the first weight error matrix and a mask matrix and a Kronecker product of the second weight error matrix and the mask matrix;
The information processing apparatus according to claim 2. The first calculator is
Calculating a first matrix product of the output from the first layer and the first weight matrix;
Calculating an output from the third layer based on a second matrix product of the first matrix product result and the second weight matrix;
The information processing apparatus according to claim 2 or 3. The result of the back propagation calculation is transmitted to another information processing apparatus in a parallel computing system including the information processing apparatus, and the result of the back propagation calculation executed in the other information processing apparatus is transmitted to the other information processing apparatus. The information processing apparatus according to claim 1, further comprising: a communication unit that receives from the information processing apparatus. The number of edges between each unit of the first layer and the second layer, and the number of edges between each unit of the third layer and the second layer are (n (n is a natural number) ) +1),
The number of units of the second layer is (n ² + n + 1),
The information processing apparatus according to any one of claims 1 to 5. Computer
Performing a forward propagation calculation of a multi-layer neural network including a first layer, a second layer receiving an output from the first layer, and a third layer receiving an output from the second layer ,
Performing back propagation calculations of the multilayer neural network based on the results of the forward propagation calculations;
Execute the process,
The output from each unit of the first layer is input to all units of the third layer via the unit of the second layer,
The number of units of the second layer includes the number of edges between each unit of the first layer and the second layer and between each unit of the third layer and the second layer. More than the number of edges,
Information processing method. On the computer,
Performing a forward propagation calculation of a multi-layer neural network including a first layer, a second layer receiving an output from the first layer, and a third layer receiving an output from the second layer ,
Performing back propagation calculations of the multilayer neural network based on the results of the forward propagation calculations;
Let the process run,
The output from each unit of the first layer is input to all units of the third layer via the unit of the second layer,
The number of units of the second layer includes the number of edges between each unit of the first layer and the second layer and between each unit of the third layer and the second layer. More than the number of edges,
program.

Images