JP6227052B2 - Processing apparatus, determination method, and program - Google Patents

Description

  The present invention relates to a processing device, a determination method, and a program.

As is well known, a neural network has learning ability, is excellent in non-linearity and pattern matching performance, and is used in many fields such as control, prediction and diagnosis.
In addition, many structures have been proposed for neural networks, but most of them that have been put to practical use are of the hierarchical type, particularly the three-layer type. Hierarchical neural networks are learned by an algorithm commonly called a back-propagation method (error back propagation method), and the internal coupling state (weight between nodes) is adjusted. Thus, when the same input data as the learning data is given, the output is almost the same as the learning data. Further, when an input close to the learning data is given, an output close to the learning data is obtained.

When dealing with a complex problem with a hierarchical neural network, the number of nodes and the number of layers in the intermediate layer are increased, which increases the amount of computation. As a solution to this problem, there is an example in which the amount of calculation is reduced by loosely coupling between nodes, and there are the following two cases as typical patent documents.
In Patent Document 1, for a plurality of input nodes, a statistical index of maximum, minimum, average, and standard deviation of learning data is used, or a correlation coefficient between input nodes or input and output of learning data is used. A group of input nodes having similar characteristics is formed, and a structure having a loosely coupled portion is formed by coupling the input node and the intermediate node in the group.
Further, in the neural network structure optimization method described in Patent Document 2, a plurality of neural networks having different structures are generated by deleting connections between arbitrary nodes, and evaluation values of the respective neural networks are calculated to obtain evaluation values. Thus, the neural network having the optimal structure is changed.

JP 2011-54200 A JP-A-9-91263

  In the conventional techniques represented by Patent Documents 1 and 2, prior learning for forming loose coupling is required before performing discriminator learning, and prior learning must be performed each time learning data is changed or modified. In other words, it takes a lot of time and computation to form a loose coupling. For this reason, there has been a problem that the speed of the discriminator learning and the discrimination processing cannot be increased.

  The present invention has been made in order to solve the above-described problems. In a hierarchical neural network, it is possible to speed up discriminator learning and discrimination processing by forming loose coupling without depending on learning data. An object of the present invention is to obtain a processing device, a determination method, and a program that can perform the above processing.

The processing apparatus according to the present invention includes a weight value learned between loosely coupled nodes and an input signal in a neural network in which a coupling portion and a non-coupling portion between nodes are formed based on a parity check matrix of an error correction code. And a processing unit for solving the problem.

  According to the present invention, since the portion where the nodes are not coupled is formed based on the parity check matrix of the error correction code, the discrimination processing can be performed at high speed. Thereby, for example, in a hierarchical neural network, it is possible to speed up the discriminator learning and the discrimination processing by forming loose coupling without depending on the learning data.

1 is a block diagram showing a configuration of a hierarchical neural network device according to Embodiment 1 of the present invention. FIG. 1 is a diagram illustrating a structure of a hierarchical neural network according to Embodiment 1. FIG. It is a figure which shows the structure of the conventional hierarchical neural network. 3 is a flowchart showing weight learning processing in the first embodiment. It is a figure which shows the test matrix of a pseudorandom code. It is a figure which shows the number of 1 and the ratio of 1 with respect to the number of rows or the number of columns of the check matrix of a Euclidean geometric code. It is a figure which shows the number of 1 and the ratio of 1 with respect to the number of rows or the number of columns of a check matrix of a projection geometric code. It is a figure which shows the number of 1 and the ratio of 1 with respect to the number of rows or the number of columns of a check matrix of a difference set cyclic code. It is a figure which shows the check matrix of a space coupling type code. It is a figure which shows the example of the connection number between the input node in the neural networks A and B, and an intermediate node. It is a block diagram which shows the structure of the hierarchical neural network apparatus concerning Embodiment 2 of this invention. 6 is a diagram illustrating a structure of a deep neural network in a second embodiment. FIG. It is a figure which shows the structure of the conventional deep neural network. 10 is a flowchart showing weight learning processing by prior learning and adjustment of weights in the second embodiment. It is a figure which shows the outline | summary of the weight learning process by the prior learning and adjustment of a weight in the case of N = 5. It is a flowchart which shows the prior learning process of a weight. It is a flowchart which shows the adjustment process of a weight.

Hereinafter, in order to describe the present invention in more detail, modes for carrying out the present invention will be described with reference to the accompanying drawings.
Embodiment 1 FIG.
1 is a block diagram showing a configuration of a hierarchical neural network apparatus according to Embodiment 1 of the present invention. In FIG. 1, a hierarchical neural network device 1 is a device that performs discrimination using a hierarchical neural network, and includes a discriminator learning unit 2, a weight storage unit 3, a learning data storage unit 4, and a teacher data storage unit 5. It is prepared for.
The hierarchical neural network includes an input layer, an intermediate layer, and an output layer, and each layer has a plurality of nodes. Hierarchical neural networks also have various problems (classification problems) by adjusting the connection state between nodes by setting arbitrary weights between the nodes of the input layer and the intermediate layer and between the nodes of the intermediate layer and the output layer. Alternatively, it functions as a discriminator that can solve a regression problem.

The discriminator learning unit 2 learns a hierarchical neural network and performs discrimination using the learned hierarchical neural network. As its configuration, a weight learning unit 20 and a discrimination processing unit 21 are provided.
The weight learning unit 20 generates a loosely coupled portion by coupling between some nodes in the hierarchical neural network based on the error correction code check matrix, and learns the weight between the coupled nodes. That is, the weight learning unit 20 uses the discrimination result output from the discrimination processing unit 21, the weight between nodes read from the weight storage unit 3 (the weight of the discriminator), and the teacher data read from the teacher data storage unit 5. When input, weight learning is performed using these data.
Also, the weight learning unit 20 assigns a node of one layer to the row element in the parity check matrix of the error correction code, assigns a node of the other layer to the column element, and joins the nodes in which 1 is set in the matrix element However, nodes whose matrix elements are 0 are not connected. Thereby, a loosely coupled portion can be generated between nodes without using learning data.

  The discrimination processing unit 21 solves the classification problem or the regression problem using a hierarchical neural network in which the weights between the combined nodes are updated with the weight values learned by the weight learning unit 20. For example, when the discrimination processing unit 21 receives the initialized weight or the learning weight from the weight storage unit 3 and inputs the learning data from the learning data storage unit 4, the discrimination processing unit 21 uses the discrimination result using these as the weight learning unit 20. Output to. The discrimination processing unit 21 receives learned weights from the weight storage unit 3 and inputs discrimination data, and outputs a discrimination result using these to a transmission device such as a display outside the device.

In the discrimination processing unit 21, learning data or discrimination data becomes an input signal between nodes of the input layer and the intermediate layer in the hierarchical neural network, and a weight between the nodes is multiplied. The sum of the multiplication results at the nodes in the intermediate layer is calculated by the threshold function and output. Here, the threshold function is f (), the output value of the threshold function of the j-th node of the intermediate layer is H _j , the input signal of the i-th node of the input layer is X _i, and the i-th node of the input layer When the weight between the j-th node of the intermediate layer is W _ji and the output value of the threshold function is H _j, it can be expressed by the following equation (1).
H _j = f (ΣX _i W _ji ) (1)

Also, between the nodes of the intermediate layer and the output layer, the output signal calculated by the threshold function is used as an input signal, and the weight between the nodes is multiplied. The sum of the multiplication results at each node in the output layer is calculated by the threshold function and output as a discrimination result. Here, when the output value of the threshold function of the k-th node of the output layer is O _k and the weight between the j-th node of the intermediate layer and the k-th node of the output layer is W _kj , the output value O _k can be represented by the following formula (2). However, examples of the threshold function f () include a sigmoid function, a tanh function, and a max function. Further, the multiplication of the weight between the nodes is performed only for a portion where there is a connection between the nodes.
O _k = f (ΣX _j W _kj ) (2)

  The weight storage unit 3 is a storage unit that stores weights between nodes in the hierarchical neural network. The weight storage unit 3 stores initial values of weights between all nodes of the hierarchical neural network during weight initialization processing, and stores weight learning values between nodes having connections when generating loose coupling. The discrimination processing unit 21 constructs a hierarchical neural network by reading each node and the weight value between the nodes from the weight storage unit 3, and uses this to solve the classification problem or the regression problem.

The learning data storage unit 4 is a storage unit that stores learning data. The learning data is data indicating state information and feature amounts for which normality and abnormality have been determined in advance. The learning data includes a case where the teacher data is known (supervised learning) and a case where the desired teacher data for the discrimination data is unknown (unsupervised learning).
The teacher data storage unit 5 is a storage unit that stores teacher data. Teacher data is desirable output data for discrimination data. The discrimination data is data to be discriminated.

Note that the weight learning unit 20 and the discrimination processing unit 21 are specific examples in which hardware and software collaborate, for example, when a microcomputer executes a program in which processing unique to the first embodiment is described. It can be realized as a means.
The weight storage unit 3, the learning data storage unit 4, and the teacher data storage unit 5 are, for example, a hard disk drive (HDD) device, a USB memory, and a storage media playback device mounted on a computer that functions as the hierarchical neural network device 1. It is constructed in a reproducible storage medium (CD, DVD, BD).

  FIG. 2 is a diagram showing the structure of a hierarchical neural network in the first embodiment, and FIG. 3 is a diagram showing the structure of a conventional hierarchical neural network. As shown in FIG. 3, in the conventional hierarchical neural network, the nodes of the input layer and the intermediate layer are all connected, and the nodes of the intermediate layer and the output layer are all connected. On the other hand, in the first embodiment, as shown in FIG. 2, a loosely coupled portion is formed in at least one of the coupling between the nodes of the input layer and the intermediate layer and the coupling between the nodes of the intermediate layer and the output layer. .

Next, the operation will be described.
FIG. 4 is a flowchart showing the weight learning process in the first embodiment. Details of the weight learning by the weight learning unit 20 will be described with reference to FIG.
First, the weight learning unit 20 initializes weights between all nodes in each layer of the hierarchical neural network (step ST1). Specifically, an initial value is given by a random number of −0.5 to +0.5 with respect to the weight between all nodes in each layer.

  Next, the weight learning unit 20 generates a loose coupling by performing coupling only between some nodes in the hierarchical neural network based on the error correction code check matrix (step ST2). The error correction code check matrix has a learning error equal to or smaller than that of a normal hierarchical neural network and is a sparse matrix. For example, any one of a pseudo random code, a finite geometric code, a cyclic code, a pseudo cyclic code, a low density parity check code (LDPC) code, and a spatially coupled code can be used.

Subsequently, the weight learning unit 20 calculates the correction amount of the weight between the nodes coupled at the loosely coupled portion so that the value of the evaluation function for evaluating the learning error becomes small (step ST3).
The evaluation function J can be expressed by, for example, the following formula (3). However, the output signal of the node is o, and the teacher data is t.
J = 1/2 · (ot) ² (3)

Thereafter, the weight learning unit 20 updates the weight value between the joined nodes from the previous value with the correction amount obtained in step ST3 (step ST4).
When updating of the weights between the nodes is completed, the weight learning unit 20 checks whether or not the weight learning end condition is satisfied (step ST5). Here, as the termination condition, for example, a case where the value of the evaluation function for calculating the error between the teacher data and the discrimination result input from the discrimination processing unit 21 is less than or equal to a predetermined threshold value can be considered. Moreover, the case where the learning frequency becomes more than a threshold frequency may be sufficient.

When it is determined that the end condition is satisfied and the weight learning should be ended (step ST5; YES), the weight learning unit 20 ends the weight learning. On the other hand, when the termination condition is not satisfied (step ST5; NO), the process returns to step ST3 and the above-described weight learning is repeated.
As a result, the discrimination processing unit 21 solves the classification problem or the regression problem by using the hierarchical neural network in which the weight between the nodes coupled by the loosely coupled portion is updated with the weight value learned by the weight learning unit 20. be able to.

Next, an example in which a loosely coupled portion is generated between the nodes of the input layer and the intermediate layer based on the check matrix of the pseudo random number code shown in FIG. The matrix in FIG. 5 is a 15 × 15 matrix with four 1's in each row and column. This is because the input layer nodes (hereinafter referred to as input nodes) x ₁ , x ₂ ,..., X ₁₅ assigned to the elements in each column and the intermediate layer nodes (hereinafter referred to as “input nodes”). , referred to as intermediate nodes) h _{1, h 2,} · · ·, there 15 respectively and h _15, four in one of the intermediate nodes as part value of the matrix element is 1 binds input Indicates that the node joins. For example, an input node x ₁ , x ₃ , x ₉ , x ₁₀ having ₁ as a matrix element is coupled to the intermediate node h ₁ .

When the number of intermediate nodes or output nodes (output layer nodes) is small (for example, 21 or less), a check matrix of a finite geometric code such as Euclidean geometric code or projective geometric code, or a difference set cyclic code, etc. Based on the check matrix of the cyclic code, if the connection is performed between nodes in which 1 is set in the matrix element as described above, a large reduction in the amount of calculation is expected. This means that the number of 1's for the number of columns or rows in the parity check matrix of Euclidean geometric code shown in FIG. 6, the parity check matrix of projective geometric code shown in FIG. 7, and the parity check matrix of difference set cyclic code shown in FIG. As is apparent from the results shown, when the number of columns or rows corresponding to the number of nodes is 21 or less, the number of 1s is significantly smaller than the number of columns or rows, and loosely coupled portions between the nodes This is because it can be formed.
The cyclic code is a code generated by cyclically shifting a code word, and has a regular arrangement. For this reason, it is suitable for learning determination of time-series data, and when it is implemented in hardware, it has a feature that it can be designed more easily than other codes.

  When there are a large number of intermediate nodes or output nodes (for example, more than 21 nodes), the number of nodes is determined based on one of the LDPC code parity check matrix, the spatially coupled code parity check matrix, and the pseudo cyclic code parity check matrix. When combining these, a significant reduction in the amount of computation can be expected. This is because, in the parity check matrix of the LDPC code, the spatially coupled code, and the pseudo cyclic code, the number of 1 included in a row and the number of 1 included in a column are 3 to 3 on average. This is because there are six. For example, even when the number of rows is 10,000, the average number of 1 included in a row is 3 to 6, and the average number of connections between nodes is 3 to 6 and loosely coupled. For this reason, the amount of calculation can be significantly reduced. In particular, since the parity check matrix of the spatially coupled code shown in FIG. 9 is a matrix in which 1s are arranged in a band shape, the effect of reducing the maximum calculation amount can be expected from the viewpoint of ease of control.

As described above, by making the nodes loosely coupled based on the parity check matrix of the error correction code, it is possible to perform discriminator learning and discrimination processing at high speed while maintaining discrimination performance.
FIG. 10 is a diagram illustrating an example of the number of connections between input nodes and intermediate nodes in the neural networks A and B. In FIG. 10, a neural network A is a normal hierarchical neural network in which all nodes are coupled, and a neural network B is a hierarchical neural network in which loose coupling is formed between nodes according to the present invention.
In the neural network B, four input nodes are connected to one intermediate node. Thus, in the first embodiment, for the neural network A, when the number of input nodes and intermediate nodes is 50, respectively, 2/25, when the number is 100, 1/25, when the number is 1000, The coupling can be reduced to 1/250. Accordingly, the product-sum operation between the input node and the intermediate node can be reduced, so that the discriminator learning and the discrimination processing can be speeded up.
In addition, since loose coupling is generated without depending on learning data, it is possible to save time and effort for prior learning even when learning data is changed or modified.

  As described above, according to the first embodiment, based on the parity check matrix of the error correction code, a connection is made between some nodes in the hierarchical neural network including the input layer, the intermediate layer, and the output layer having nodes. A weight learning unit 20 that generates a loosely coupled portion and learns weights between the coupled nodes, and a hierarchical type in which the weights between the coupled nodes are updated with the weight values learned by the weight learning unit 20 And a discrimination processing unit 21 for solving a classification problem or a regression problem using a neural network. In this way, it is possible to speed up the discriminator learning and the discrimination processing by forming loose coupling without depending on the learning data in the hierarchical neural network.

Embodiment 2. FIG.
The first embodiment has shown the case of speeding up discriminator learning and discrimination processing using a general three-layer neural network. In the second embodiment, a case will be described in which a classifier learning and a discrimination process using a deep neural network, which has been attracting attention in recent years, is an advanced form of a hierarchical neural network and is accelerated.

  FIG. 11 is a block diagram showing a configuration of a hierarchical neural network apparatus according to Embodiment 2 of the present invention. A hierarchical neural network device 1A shown in FIG. 11 is a device that solves a classification problem or a regression problem using a deep neural network, and includes a discriminator learning unit 2A, a weight storage unit 3, a learning data storage unit 4, and a teacher data storage unit. 5 is configured. The deep neural network includes an input layer, a plurality of intermediate layers, and an output layer, and each layer has a plurality of nodes. In addition, deep neural networks can solve various problems (classification problems) by adjusting the connection state between nodes by setting arbitrary weights between the nodes of the input layer and the intermediate layer and between the nodes of the intermediate layer and the output layer. Alternatively, it functions as a discriminator that can solve a regression problem.

  FIG. 12 is a diagram showing the structure of a deep neural network in the second embodiment, and FIG. 13 is a diagram showing the structure of a conventional deep neural network. As shown in FIG. 13, in the conventional deep neural network, the nodes of the input layer and the intermediate layer are all connected, the nodes of the intermediate layers are all connected, and the nodes of the intermediate layer and the output layer are all connected. ing. On the other hand, in the second embodiment, as shown in FIG. 12, at least of the coupling between the nodes of the input layer and the middle layer, the coupling between the nodes of the middle layer, and the coupling between the nodes of the middle layer and the output layer. A loosely coupled portion is formed on one side.

The discriminator learning unit 2A learns a deep neural network and solves a classification problem or a regression problem using the learned deep neural network. As its configuration, a discrimination processing unit 21, a weight pre-learning unit 22, and a weight adjustment unit 23 are provided.
The weight pre-learning unit 22 generates a loosely coupled part by coupling between some nodes in the deep neural network based on the error correction code check matrix, and learns the weight between the coupled nodes without supervision. For example, the weight pre-learning unit 22 performs weight pre-learning when an initialized weight between nodes and learning data are input.
Further, the weight pre-learning unit 22 assigns a node of one layer to the row element in the parity check matrix of the error correction code, assigns a node of the other layer to the column element, and moves between nodes where 1 is set in the matrix element. The nodes are connected, and the nodes whose matrix elements are 0 are not connected. Thereby, a loosely coupled portion can be generated between nodes without using learning data.
The weight adjustment unit 23 finely adjusts the weight learned by the weight pre-learning unit 22 by supervised learning. That is, the weight adjustment unit 23 performs fine adjustment of the weight only between the coupled nodes.

  The discrimination processing unit 21 according to the second embodiment inputs weights learned in advance from the weight storage unit 3 or learning weights, and inputs learning data from the learning data storage unit 4. Output to the adjustment unit 23. The discrimination processing unit 21 receives learned weights from the weight storage unit 3 and inputs discrimination data, and outputs a discrimination result using these to a transmission device such as a display outside the device.

In the discrimination processing unit 21, learning data or discrimination data becomes an input signal between the nodes of the input layer and the first intermediate layer in the deep neural network, and the weight between the nodes is multiplied. The sum of the multiplication results at the intermediate nodes in the first intermediate layer is calculated by the threshold function and output. Here, the threshold function is f (), the output value of the threshold function of the j-th intermediate node of the first intermediate layer is H _{1, j} , the input signal of the i-th input node is X _i , and the i-th When the weight between the input node and the j-th intermediate node of the first intermediate layer is W _{1, j, i} , the output value of the threshold function H _{1, j} is the following equation (4 ).
H _{1, j} = f (ΣX _i W _{1, j, i} ) (4)

Further, between the jth intermediate node of the intermediate layer of the n-1st layer (n is an integer of 2 or more) and the mth intermediate node of the nth intermediate layer, the n−1th layer The output signal calculated by the threshold function of the intermediate node is used as an input signal, and the weight between the nodes is multiplied. The sum of the multiplication results at each intermediate node in the nth intermediate layer is calculated by a threshold function and output. Here, the output value of the threshold function of the jth intermediate node of the (n−1) th intermediate layer is H _{n−1, j} , and the output of the threshold function of the mth intermediate node of the nth intermediate layer is The value is H _{n, m} , and the weight between the jth intermediate node of the (n−1) th intermediate layer and the mth intermediate node of the nth intermediate layer is W _{n, m, n−1. , J} , the output value H _{n, m} of the threshold function can be expressed by the following equation (5).
H _{n, m} = f (ΣH _{n−1, j} W _{n, m, n−1, j} ) (5)

Between the intermediate node and the output node, the output signal calculated by the threshold function of the intermediate node is used as an input signal, and the weight between the nodes is multiplied. The sum of the multiplication results at each output node in the output layer is calculated by the threshold function and output as a discrimination result.
Here, the output value of the threshold function of the kth output node of the output layer is O _k , and the weight between the jth intermediate node of the intermediate layer of the (N−1) th layer and the kth output node of the output layer Is W _{k, N−1, j} , the output value O _k of the threshold function can be expressed by the following equation (6).
However, examples of the threshold function f () include a sigmoid function, a tanh function, and a max function. Further, the multiplication of the weight between the nodes is performed only for a portion where there is a connection between the nodes.
O _k = f (ΣH _{N−1, j} W _{k, N−1, j} ) (6)

  Note that the discrimination processing unit 21, the weight pre-learning unit 22, and the weight adjustment unit 23 cooperate with hardware and software by, for example, a microcomputer executing a program in which processing unique to the second embodiment is described. It can be realized as a concrete means that worked.

Next, the operation will be described.
FIG. 14 is a flowchart showing the weight learning process in the second embodiment.
First, the weight pre-learning unit 22 initializes weights between all nodes in each layer of the deep neural network (step ST1a). Specifically, as in the first embodiment, initial values are given as random numbers of −0.5 to +0.5 with respect to the weights between all the nodes in each layer.

  Next, the weight pre-learning unit 22 performs a coupling between some nodes in the deep neural network based on the error correction code check matrix to generate a loose coupling (step ST2a). The error correction code check matrix has a learning error equal to or smaller than that of a normal deep neural network and is a sparse matrix. For example, any one of a pseudo random number code, a finite geometric code, a cyclic code, a pseudo cyclic code, an LDPC code, and a spatially coupled code can be used.

Subsequently, as illustrated in FIG. 15, the weight pre-learning unit 22 uses a weight between nodes (W ₁ ) having connections up to the (N−1) -th layer when the deep neural network has N layers (N is an integer). _, W _2, ..., W _N−2 ) (step ST3a).
In this pre-learning, first, in the two-layer structure of the first layer and the second layer, the weight W ₁ between the nodes of the first layer and the second layer is learned without supervision. Next, in the two-layer structure of the second layer and the third layer, the weight output between the nodes of the second layer and the third layer with the signal output from the node of the second layer in the unsupervised learning of the weight W ₁ as an input signal the W ₂ to unsupervised learning. This process is repeated until the weight W _N−2 between the nodes of the (N−2) -th layer and the (N−1) -th layer is pre-learned (see FIG. 15, when N = 5).

Details of weight prior learning will be described with reference to FIG.
First, the weight pre-learning unit 22 initially sets the signal output in the previous pre-learning as an input signal when pre-learning the weight between the second layer node and the third and higher layer nodes (step ST1b). ). Next, the weight pre-learning unit 22 calculates the correction amount of the weight between the coupled nodes so that the log likelihood increases (step ST2b).
Subsequently, the weight pre-learning unit 22 updates and corrects the weight value between the joined nodes with the calculated weight correction amount (step ST3b).

When the updating of the weights between the nodes is completed, the weight pre-learning unit 22 checks whether or not the weight pre-learning end condition for the current learning target layer is satisfied (step ST4b). Here, as the termination condition, for example, a case where the number of learning times is equal to or greater than a threshold number is considered.
When it is determined that the end condition of the current learning target layer is satisfied and the weight pre-learning should be ended (step ST4b; YES), the weight pre-learning unit 22 proceeds to the process of step ST5b.
On the other hand, when the termination condition is not satisfied (step ST4b; NO), the process returns to step ST2b, and the weight pre-learning described above is repeated.

In step ST5b, when the deep neural network is the N layer, the weight pre-learning unit 22 determines whether or not the weight pre-learning between the nodes coupled in all layers up to the (N-1) th layer is completed. Check. When the weight pre-learning of all layers is not completed (step ST5b; NO), the process returns to step ST1b, and the weight pre-learning described above is performed with the next layer (the upper layer) as a learning target. If the weight pre-learning for all layers has been completed (step ST5b; YES), the weight pre-learning unit 22 ends the pre-learning. Here, deep neural network for N layer, prior learning of N-2 pieces of weights from W ₁ W _N-2 is performed.

Returning to the description of FIG.
When the weight pre-learning unit 22 completes the weight pre-learning, the weight adjusting unit 23 finely adjusts the weight pre-learned by the weight pre-learning unit 22 through supervised learning and performs optimization (step ST4a). The details of the fine adjustment of the weight will be described below with reference to FIG.
First, the weight adjustment unit 23 performs supervised learning using the teacher data read from the teacher data storage unit 5 so that the value of the evaluation function J for evaluating the learning error as shown in the above equation (3) becomes small. Thus, the weight pre-learning unit 22 optimizes the weight between the nodes pre-learned to calculate the weight correction amount (step ST1c).

Next, the weight adjustment unit 23 updates the value of the weight between the nodes pre-learned by the weight pre-learning unit 22 with the correction amount obtained in step ST1c (step ST2c).
When the updating of the weight between the nodes is completed, the weight adjusting unit 23 checks whether or not the condition for ending the fine weight adjustment is satisfied (step ST3c). Here, as the termination condition, for example, a case where the value of the evaluation function for calculating the error between the teacher data and the discrimination result input from the discrimination processing unit 21 is less than or equal to a predetermined threshold value can be considered. Moreover, the case where the learning frequency becomes more than a threshold frequency may be sufficient.

  When it is determined that the end condition is satisfied and fine adjustment of the weight should be ended (step ST3c; YES), the weight adjustment unit 23 ends the fine adjustment of the weight. On the other hand, when the termination condition is not satisfied (step ST3c; NO), the process returns to step ST1c, and the above-described fine adjustment of the weight is repeated. Thereby, the discrimination processing unit 21 is pre-learned by the weight pre-learning unit 22 for the weight between the nodes coupled by the loosely coupled portion, and is updated with the weight value optimized and adjusted by the weight adjusting unit 23. A classification problem or regression problem can be solved using a network.

In the deep neural network, when the number of intermediate nodes or output nodes is small (for example, 21 or less), as in the first embodiment, a parity check matrix of a finite geometric code such as a Euclidean geometric code or a projective geometric code, or In a check matrix of a cyclic code such as a difference set cyclic code, a node of one layer is assigned to a row element, a node of the other layer is assigned to a column element, and a connection is made between nodes in which 1 is set in the matrix element. A significant reduction in the amount of computation is expected. This is 1 for the number of columns or rows in the parity check matrix of Euclidean geometric code shown in FIG. 6, the parity check matrix of projective geometric code shown in FIG. 7, and the parity check matrix of difference set cyclic code shown in FIG. As is clear from the results of the number of nodes, when the number of columns or rows corresponding to the number of nodes is 21 or less, the number of 1 is much smaller than the number of columns or rows, and the nodes are loosely coupled. This is because the portion can be formed.
The cyclic code is a code generated by cyclically shifting a code word, and has a regular arrangement. For this reason, it is suitable for learning determination of time-series data, and when it is implemented in hardware, it has a feature that it can be designed more easily than other codes.

  Further, in the deep neural network, when the number of intermediate nodes or output nodes is large (for example, more than 21 nodes), the LDPC code parity check matrix, the spatially coupled code parity check matrix, and the pseudo cyclic code as in the first embodiment If the connection between nodes is performed based on one of the check matrices, a large reduction in the amount of computation can be expected. This is because LDPC codes, spatially coupled codes, and quasi-cyclic code parity check matrices do not depend on the number of rows or the number of columns, and the number of 1 included in a row or the number of 1 included in a column is 3 to 6 on average. This is because it becomes an individual. For example, even when the number of rows is 10,000, the average number of 1 included in a row is 3 to 6, and the average number of connections between nodes is 3 to 6 and loosely coupled. For this reason, the amount of calculation can be significantly reduced. In particular, since the parity check matrix of the spatially coupled code shown in FIG. 9 is a matrix in which matrix elements 1 are arranged in a band shape, the effect of reducing the maximum calculation amount can be expected from the viewpoint of ease of control.

  As described above, according to the second embodiment, based on the parity check matrix of the error correction code, between some nodes in a deep neural network composed of an input layer having nodes, a plurality of intermediate layers, and an output layer. A weight pre-learning unit 22 that generates a loosely coupled part by performing coupling, learns the weight between the coupled nodes without supervision, and a weight adjustment unit 23 that adjusts the weight learned by the weight pre-learning unit 22 by supervised learning And a discrimination processing unit 21 that solves the classification problem or the regression problem using a deep neural network in which the weight between the coupled nodes is updated with the weight value adjusted by the weight adjustment unit 23. Thus, by forming loose coupling in the deep neural network, it is possible to speed up the discriminator learning and the discrimination processing. In particular, the deep neural network has a larger number of intermediate layers than the hierarchical neural network shown in the first embodiment and has many places where a loosely coupled portion can be formed. large. In addition, since loose coupling is generated without depending on learning data, it is possible to save the trouble of performing prior learning when learning data is changed or modified.

  In the present invention, within the scope of the invention, any combination of each embodiment, any component of each embodiment can be modified, or any component can be omitted in each embodiment. .

  The hierarchical neural network device according to the present invention can speed up discriminator learning and discrimination processing by forming loose coupling without depending on learning data in the hierarchical neural network. It can be applied to information processing related to prediction and diagnosis.

  1, 1A hierarchical neural network device, 2, 2A discriminator learning unit, 3 weight storage unit, 4 learning data storage unit, 5 teacher data storage unit, 20 weight learning unit, 21 discrimination processing unit, 22 weight pre-learning unit, 23 Weight adjustment unit.

Claims (7)

Processing to solve problems using weight values learned between loosely connected nodes and input signals in a neural network in which connected and unconnected portions between nodes are formed based on a parity check matrix of error correction codes The processing apparatus provided with a unit. A storage unit for storing weight information of a neural network in which a connection unit and a non-connection unit between nodes are formed based on a parity check matrix of an error correction code;
A processing apparatus comprising: a discrimination processing unit that calculates and outputs a discrimination result for an input signal using weight information of the neural network.   The processing apparatus according to claim 2, wherein the neural network is a deep neural network.   4. The processing apparatus according to claim 1, wherein the check matrix includes a node of one layer in the hierarchical neural network as a row element and a node of another layer as a column element. 5. .   The processing apparatus according to claim 1, wherein the input signal is time-series data, and the error correction code is a cyclic code. Based on the error correction code check matrix, the weight value of the coupling unit coupled to one node is read out from a storage unit that stores information of the neural network in which the coupling unit and the non-coupling unit between the nodes are formed. Calculating a multiplication result of the weight value and an input signal of another node coupled by the coupling unit for a plurality of the coupling units;
Outputting a discrimination result using the multiplication result calculated for the plurality of coupling units;
Method of determination with Based on the error correction code check matrix, the weight value of the coupling unit coupled to one node is read out from a storage unit that stores information of the neural network in which the coupling unit and the non-coupling unit between the nodes are formed. Calculating a multiplication result of the weight value and an input signal of another node coupled by the coupling unit for a plurality of the coupling units;
Outputting a discrimination result using the multiplication result calculated for the plurality of coupling units;
A program that causes a computer to execute.

Images