JP2016218513A - Neural network and computer program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a neural network with high generalization performance, by which prediction is possible in short time.SOLUTION: A neural network 80 comprises an input layer 90 and a hidden layer 92 for receiving output from the input layer 90. The hidden layer 92 includes a plurality of hidden layer units 100 and 102 each of which is connected with a plurality of input nodes of the input layer 90. Each of the plurality of hidden layer units 100 and 102 includes: a plurality of COS elements 120-128 each of which has input connected with the plurality of input nodes with weighting, and uses a cosine function as an activation function; and a Σ element 140 which is connected so as to receive the output of the plurality of COS elements 120-128 in the same hidden layer with weighting, and uses a subdifferentiable function as the activation function. The neural network 80 may further include an output layer 94 connected so as to receive the output of the hidden layer units 100 and 102.SELECTED DRAWING: Figure 3

Description

  The present invention relates to machine learning, and more particularly, to a neural network that can perform inference with high accuracy even on a computer with small calculation resources.

  Machine learning is a learning method based on inductive reasoning that aims to acquire general rules or trends from a limited amount of cases. Mathematically, it is formulated as a global estimation of the source data distribution, assuming that the case (training) data is sampled from the true data distribution. The data distribution can be understood as a continuous function having a data space as a domain and a space expressing a data quality such as probability as a range. One method for approximating this continuous function from training data is a neural network. Neural networks are widely used as multi-layered perceptrons (MLPs) or feed forward neural networks (FFNNs) in various identification tasks such as language models, speech recognition, and image recognition.

  Any continuous function or data distribution can be represented in a two-layer neural network using a logistic sigmoid function as an activation function. However, in a network with a small number of layers such as about two layers, there is a tendency that only a local distribution such as individual training data points is learned, and overfitting that cannot acquire a global distribution occurs. This is shown in FIG. FIG. 1 assumes a case where two types of data exist in a two-dimensional data space and their training data (indicated by x and ◯) are obtained. When these two types of data are identified as a neural network task, it is necessary to obtain an identification curve for identifying these two types of data in the original data space. However, if too many parameters are assumed, the identification curve often overfits the training data. If the discrimination curve obtained by overfitting is, for example, the discrimination curve of FIG. 1, this discrimination curve 30 has very high discrimination performance for training data and sometimes gives a complete discrimination result, but discrimination performance for data other than training data. Conversely, the (generalization performance) deteriorates. On the other hand, the identification curve 32 obtained by proper training does not always give a complete identification result for all test data, but data with relatively high accuracy is provided even when data other than training data is given. Can be identified.

  In order to suppress overfitting, a technique called regularization (Non-Patent Document 1, Non-Patent Document 2) is widely used in general machine learning. Regularization is a method of reducing the number of parameters that affect approximation or reducing the value of parameters when the identification model learns to approximate training data. This approach is based on the assumption that the original distribution from which training data is extracted has a smooth shape. That is, regularization is based on the assumption that the original distribution is a function composed of a small number of small value parameters. In general, by adding the L2 norm or L1 norm of the parameter to the objective function to be minimized, it is used for the purpose of setting the value of many parameters to 0 or near 0. That is, the generalization performance is improved by imposing a restriction that substantially restricts the number of parameters used for inference by regularization.

  Along with regularization, there is an idea of improving generalization performance from the viewpoint of the calculation structure (function definition) of the model. As one of the methods, there is a multilayered neural network called deep learning. In a neural network consisting of many layers, it is said that local data distribution is captured in layers close to the input, and global distribution is captured by a combination of local data distributions as it proceeds to the output layer. (Non-patent Document 3). In some identification tasks such as image recognition, generalization performance that far surpasses conventional techniques is achieved by deep learning (Non-Patent Document 4).

  FIG. 2 shows a configuration example of a multilayered neural network (deep neural network: DNN) 50. Referring to FIG. 2, DNN 50 includes an input layer 52, an intermediate layer 54 including a plurality of hidden layers, and an output layer 56. In this example, the intermediate layer 54 includes five hidden layers. The back propagation method is used for learning the DNN 50. That is, training data is given to the input layer 52, and a predicted output for the training data is obtained from the output layer 56 via the intermediate layer 54. The error between the output and the correct data is given from the output side, and the weight and bias of the connection between the nodes are updated so as to minimize a predetermined error function.

  Conventionally, logistic sigmoid functions, hyperbolic tangents, and the like have been widely used as activation functions for each layer. However, recently, on the premise of multilayering, various functions (Non-Patent Document 5 and Non-Patent Document 6) have been proposed for the activation function of each layer, and these have been shown to improve generalization performance. .

Evgeniou, T., Pontil, M., & Poggio, T. (2000). Regularization networks and support vector machines.Advances in computational mathematics, 13 (1), 1-50. Girosi, F., Jones, M., & Poggio, T. (1995). Regularization theory and neural networks architectures. Neural computation, 7 (2), 219-269. Montufar, G., Pascanu, R., Cho, K. and Bengio, Y. (2014) .On the Number of Linear Regions of Deep Neural Networks.http: //arxiv.org/abs/1402.1869 Schmidhuber, J. (2015). Deep Learning in Neural Networks: An Overview. Neural Networks, Vol 61, pp 85-117. (Http://arxiv.org/abs/1404.7828) Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier networks.Proceedings of the 14th International Conference on Artificial Intelligence and Statistics.JMLR W & CP Volume.Vol. 15. Goodfellow, I., Warde-farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013) .Maxout Networks.In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 1319-1327.

  As described above, improvement in generalization performance by multilayering has been shown by the prior art. However, there is a problem that the calculation amount has increased at the same time. Therefore, a large-scale computing device is still necessary for improving the performance. In addition, the computation efficiency of each layer has improved due to the progress of massive parallelization of computer hardware, such as the spread of general-purpose GPU computers. However, in the multilayer model, the calculation of the next layer is not possible until the computation on the input side layer is completed. There is a restriction that it is possible. Therefore, there is a problem that the waiting time is inevitably increased in proportion to the increase in the number of layers due to the multi-layering, and the calculation efficiency is reduced not only during training but also during prediction.

  Accordingly, an object of the present invention is to provide a neural network with high generalization performance that can be predicted in a short time.

  The neural network according to the first aspect of the present invention includes an input layer having a plurality of input nodes and a nonlinear function layer having an input connected to receive an output from the input layer. The nonlinear function layer includes a plurality of hidden layers, each connected to receive outputs from a plurality of input nodes of the input layer, each capable of approximating an arbitrary independent component. Each of the plurality of hidden layers has an input connected to each of a plurality of input nodes with weights, and is the same as a plurality of nerve cell elements that use a sub-differentiable periodic function as an activation function. And an output aggregating element that is connected so as to receive weighted outputs of a plurality of nerve cell elements in the hidden layer and uses a subdifferentiable function as an activation function.

  Preferably, the periodic function is a cosine function or sine function or a combination thereof, or a cosine function or sine function approximated by piecewise linear approximation.

  More preferably, the neural network is trained using L2 regularization for the weight parameter on the input side of each of the plurality of hidden layers.

  The neural network may be trained using a quasi-Newton method.

  More preferably, the activation function of the output aggregation element may be an identity function, that is, a function that outputs the same value as a given input.

  The neural network may further include an output layer connected to receive the output of the plurality of hidden layer output aggregating elements. The output layer includes a plurality of output nerve cell elements each connected to receive the output of a plurality of hidden layer output aggregation elements with weights.

  The activation function of the output aggregation element may be a softmax function.

  A computer program according to the second aspect of the present invention causes a computer to function as any one of the neural networks described above.

It is a figure which shows the relationship between the data distribution in a two-dimensional space, and an identification function for demonstrating the problem of overfitting in machine learning. It is a schematic diagram of the neural network for demonstrating the problem of the conventional deep neural network. It is a figure which shows typically schematic structure of the neural network which concerns on one embodiment of this invention. It is a graph which shows an example of an objective function when a regular term is not added to an error. It is a graph which shows an example of the objective function when a regular term is added to an error. It is a block diagram of the speech recognition apparatus using the acoustic model by the neural network which concerns on one embodiment of this invention. In one embodiment of this invention, it is a graph which shows the improvement of the generalization performance in the handwritten character recognition when regularizing. In 1 embodiment of this invention, it is a graph which shows the generalization performance in the handwritten character recognition when not regularizing. 1 is an external view of a computer system that implements a character recognition device according to an embodiment of the present invention. It is a block diagram which shows the hardware constitutions of the computer shown in FIG.

  In the following description and drawings, the same parts are denoted by the same reference numerals. Therefore, detailed description thereof will not be repeated. In the following description of the embodiment, the neural network is referred to as “NN”.

[Basic concept]
In machine learning, as a framework for obtaining a continuous function that approximates training data, a network structure that represents a frequency domain in a discrete Fourier transform of a continuous function is used in one embodiment of the present invention described below.

  FIG. 3 shows the structure of the NN used in the present embodiment, taking a function approximating two label distributions as an example. Referring to FIG. 3, NN 80 includes an input layer 90, a hidden layer 92 connected to receive the output of input layer 90, and an output layer 94 connected to receive the output of hidden layer 92.

  In this example, input layer 90 includes three nodes. Each node is called a neural cell element (neuron). Each neuron outputs a value of a predetermined activation function having the input weighted sum as an input. The activation function may be a monotonically increasing function, but the activation function used in the present embodiment will be described later.

  The hidden layer 92 includes two hidden layer units 100 and 102. Both hidden layer units 100 and 102 have the same structure. Accordingly, only the hidden layer unit 100 will be described below as an example. The hidden layer 92 is only one layer in terms of the number of layers. In FIG. 3, the number of hidden layer units is two for the sake of explanation, but there are about ten in this embodiment.

  The hidden layer unit 100 has inputs coupled to the outputs of the three input elements of the input layer 90, and COS elements 120, 122, 124, 126, and 128 that use a cosine function as an activation function, and a COS element 120. , 122, 124, 126 and 128 are coupled to these to receive the outputs and include a Σ element 140 using the identity map as the activation function. Each of these elements is a kind of nerve cell element.

  In this embodiment, output layer 94 includes nodes coupled to both so as to receive the outputs of hidden layer units 100 and 102.

  Weights and biases are set for the coupling between the elements. The output of each Σ element is expressed by the following equation.

上式のベクトルｘは入力ベクトルを表し、行列ａ_ｄは入力層９０及び隠れ層ユニット１００、１０２の間の素子間の結合の重み行列を表し、ベクトルｂ_ｄは隠れ層ユニット１００及び１０２の各素子の出力に加算されるバイアスベクトルを表し、ベクトルｃ_ｄは隠れ層ユニット１００、１０２の各ＣＯＳ素子１２０、１２２、１２４、１２６及び１２８とΣ素子１４０との間の結合の重みベクトルを表す。これら重み及びバイアスを訓練データに基づいて決定する過程が学習である。学習の結果、各Σ素子は、入力層９０に対する入力について任意の連続関数を近似可能である。各Σ素子が近似する連続関数の周波数領域において、ＣＯＳ素子の入力側の重みは周波数成分に対応し、同入力側のバイアスは位相成分に対応し、同出力側の重みは振幅成分に対応する。各Σ素子は、互いにＣＯＳ素子の出力を共有しないことにより、入力空間上の任意の連続関数を独立に近似可能である。

The vector x in the above equation represents the input vector, the matrix a _d represents the weight matrix of the coupling between the elements between the input layer 90 and the hidden layer units 100 and 102, and the vector b _d represents each of the hidden layer units 100 and 102. represents a bias vector being added to the output of the device, the vector _{c d} represents the weight vector of the bond between each COS elements 120, 122, 124, 126 and 128 and Σ element 140 of the hidden layer units 100 and 102. Learning is a process of determining these weights and biases based on training data. As a result of learning, each Σ element can approximate an arbitrary continuous function with respect to the input to the input layer 90. In the frequency domain of the continuous function approximated by each Σ element, the weight on the input side of the COS element corresponds to the frequency component, the bias on the input side corresponds to the phase component, and the weight on the output side corresponds to the amplitude component. . Each Σ element can independently approximate any continuous function in the input space by not sharing the output of the COS element.

  The output layer 94 can be omitted by preparing the same number of hidden layer units 100 and 102 as the number of labels. Further, by making the activation function of the Σ element of each hidden layer unit a softmax function, the output can be used as a probability value. The softmax function is a function (i = 1,..., n) represented by the following expression.

  By preparing the output layer 94 as shown in FIG. 3, it is possible to incorporate more or fewer Σ elements than the number of labels. Note that the approximate performance of the network is equivalent even when a sine function is used instead of a cosine function in a COS element. Further, differentiable periodic functions other than these or combinations thereof may be used.

  Also in this embodiment, learning using the error back-propagation method is performed as in the conventional FFNN. However, in this embodiment, the cosine function is used as the activation function in the COS element of the hidden layer unit. For this reason, as described below, it is difficult to perform optimization learning of the hidden layer units 100 and 102 on the training data using the same method as the conventional method. Therefore, in this embodiment, the following two methods are used during learning. The first is regularization, and the second is the quasi-Newton method.

  The network structure used in this embodiment uses a frequency domain representation of data distribution. Therefore, each element of the frequency component, the phase component, and the amplitude component that can be interpreted as a property in the frequency domain of the function to be approximated can be individually controlled in the NN 80. In general, the global trend of a function is expressed by low frequency components. Therefore, generalization performance can be improved by obtaining a global data distribution. Therefore, in the present embodiment, the data distribution is approximated by the low frequency region by regularizing the frequency component. The regularization term is a downward convex function (convex function) with the origin at the apex with respect to the parameter representing the frequency component. If the objective function in learning is an error between the approximate function and the training data, by adding regularization to the error, the objective function becomes a minimum value near the origin, and becomes a convex function globally. 4 and 5 show examples of the shape of the objective function relating to the value of the frequency parameter in the network used in the present embodiment. An example of the shape of the objective function without regularization is shown in FIG. FIG. 5 shows an example of the shape of the objective function when the L2 norm of the parameter representing the frequency component is added to the objective function to be minimized (L2 regularization).

  4 and 5 are compared, it can be seen that there are a plurality of minimum values in FIG. 4, whereas there is only one global minimum value near the origin in FIG. Therefore, each parameter is learned so that the error function converges to the global minimum value. However, as is apparent from FIG. 5, there are a plurality of local minimum values (minimum values) in this graph. In the normal learning process, there is a risk that the parameter will converge to one of these minimum values. Such a situation must be avoided. Therefore, in the present embodiment, a global optimization method such as a quasi-Newton method including the L-BFGS method is further used for the frequency parameter. The quasi-Newton method has a feature that it is difficult to converge to a local optimal solution. In particular, it is known that the L-BGFS method requires less memory. Therefore, the apparatus can be used even with a device having a small calculation resource.

[Constitution]
FIG. 6 is a block diagram showing a character recognition device 180 and a character recognition NN learning device 186 that learns the NN of the character recognition device 180 as an example of employing the NN having the above-described configuration.

  Referring to FIG. 6, character recognition device 180 receives input image 182 including handwritten characters, performs handwritten character recognition, and outputs character recognition text 184. In this example, it is assumed that the character recognition device 180 receives an input image 182 from another scanner or storage device (not shown). The character recognition device 180 receives the input image 182 and temporarily stores the input image 182. After correcting the inclination of the input image stored in the input image storage device 200, the individual character image An image correction unit 202 that normalizes each character image, an image storage unit 204 that stores the character image for each character, and a learned character recognition that includes the NN according to the present embodiment. The NN 208 includes a decoder 210 that reads a character image for each character from the image storage unit 204, inputs the character image to the character recognition NN 208, and obtains the result to output the character recognition text 184.

  In the character recognition NN 208, learning by the character recognition NN learning device 186 is performed in advance. The character recognition NN learning device 186 uses a method similar to that of the image correction unit 202 from a handwritten character database (DB) 240 that stores an image of handwritten characters together with a character label of the character, and handwritten characters stored in the handwritten character DB 240. A learning data generation unit 242 that corrects a character image and generates learning data paired with a character label, a learning data storage device 244 that stores learning data generated by the learning data generation unit 242, and a learning data storage device 244 And a learning processing unit 246 for learning the character recognition NN 208 using the above-described method. The number of nodes in the input layer of the character recognition NN 208 is equal to the number of elements in the learning data excluding the character label. The character label is used as teacher data at the time of learning to generate a correct vector used for back propagation of error from the output layer side of the character recognition NN 208.

[Operation]
The character recognition NN learning device 186 and the character recognition device 180 described above operate as follows. First, the character recognition NN 208 is learned by the character recognition NN learning device 186, and then the character recognition of the input image 182 by the character recognition device 180 is performed.

  In learning, the learning data generation unit 242 reads a handwritten character image stored in the handwritten character DB 240, corrects and normalizes the image in the same manner as the image correction unit 202, and generates learning data in combination with a character label. The generated learning data is stored in the learning data storage device 244. The learning processing unit 246 uses the learning data stored in the learning data storage device 244 to learn the character recognition NN 208 using the method described above. When learning of the character recognition NN 208 by the learning processing unit 246 is completed, the character recognition of the input image 182 by the character recognition device 180 becomes possible.

  In character recognition, an input image 182 is generated by a scanner or the like (not shown) and stored in the input image storage device 200. The image correction unit 202 corrects the image by performing image inclination correction, character region extraction, character region normalization, and the like, and stores the image in the image storage unit 204. The decoder 210 reads this image from the image storage unit 204 and gives it to the character recognition NN 208 as an input. As a result of learning, the character recognition NN 208 returns an output specifying the character label corresponding to the input character image to the decoder 210. The decoder 210 generates a character recognition text 184 from the character label and outputs it.

[Experimental result]
In order to investigate the performance of the character recognition NN 208, an experiment on handwritten character recognition was conducted. In the experiment, an MNIST handwritten character data set was used. The performance according to the present embodiment, the maxout model shown in Non-Patent Document 6 as the NN identification model, and the performance based on the rectifier shown in Non-Patent Document 5 are shown together in Table 1 below.

なお、表１において例えば「実施の形態（３０／１０）」の「３０」は隠れ層ユニット数を、「１０」は隠れ層ユニットのＣＯＳ素子数を、それぞれ表す。他も同様である。

In Table 1, for example, “30” in “Embodiment (30/10)” represents the number of hidden layer units, and “10” represents the number of COS elements of the hidden layer units. Others are the same.

  As shown in Table 1, the identification performance of the NN according to the present embodiment does not reach that of the latest method. However, the performance is not so bad. Of particular note is the small number of parameters. In the embodiment (30/10), the number of parameters is 472K. On the other hand, in maxout, the number of parameters is 1233K, and in rectifier is 3,798K. Thus, according to the present embodiment, a relatively high accuracy can be obtained with a small number of parameters. Furthermore, in the embodiment (20/10), the number of parameters is 157K, and in the embodiment (10/15), the number of parameters is 118K. Nevertheless, the discrimination error rate increases only slightly and the degradation of the discrimination performance is negligible.

  Furthermore, in order to confirm how much regularization contributes to the improvement of generalization performance, when regularization is performed and learning of the character recognition NN 208 is performed, and when learning is performed without regularization The results were compared. FIG. 7 shows a case where regularization is performed, and FIG. 8 shows a case where regularization is not performed. In both figures, the solid line 260 and 280 indicate the discrimination error rate for the training data, and the broken lines 264 and 284 indicate the discrimination error rate for the test data. In the graph, the horizontal axis indicates the number of learning repetitions, and the vertical axis indicates the identification error rate at that time.

  As indicated by a solid line 280 in FIG. 8, when regularization is not performed, the identification error for the test data greatly decreases with learning. However, as indicated by the broken line 284, the accuracy for the test data was hardly increased. On the other hand, as shown in FIG. 7, when regularization is performed, the decrease in the error rate with respect to the learning data does not reach when the regularization is not performed as indicated by the solid line 260, but the broken line 264. As shown by, the discrimination error rate for the test data decreased with learning. That is, identification errors are reduced and the identification accuracy is increased. As a result, it can be said that the generalization performance is higher than the case where regularization is not performed.

[Effect of the embodiment]
As described above, according to the above embodiment, the generalization performance can be improved without requiring multi-layering. Therefore, the ratio of the parallel calculation in the entire calculation increases, and the calculation efficiency when using the parallel computer is improved. Further, the waiting time in each layer is reduced by reducing the number of layers, so that the calculation time at the time of prediction and the computer resources necessary for the calculation are reduced.

  Since the Σ element, which is the calculation unit of the intermediate layer, does not share the output of the COS element as its input, the number of paths in error back propagation during training is reduced. In other words, the credit allocation problem is eased. Therefore, the NN of the above embodiment can be efficiently optimized as compared with the structure of all bonds, and has the effect of reducing the training time.

  Conventional regularization has been aimed at reducing the number of parameters expressing the distribution, so the parameters involved in the identification have remained part of the network. On the other hand, in the above-described embodiment, a periodic function called cosine or sine that continues to change globally is used as the activation function. Therefore, all elements affect the identification result. As a result, since the number of parameters prepared in advance is expected to be reduced, the overall calculation scale is reduced, and high-performance discrimination on a small computer becomes possible. In the above embodiment, a cosine function is used as the periodic function. However, as described above, a sine function may be used. A combination of a cosine function and a sine function may be used. Further, a function obtained by piecewise linear approximation of a cosine function or sine function (a cosine function or sine function obtained by piecewise linear approximation) similarly becomes a periodic function, and can be used in place of the cosine function or sine function. In the case of using a piecewise linear approximation function, training using the quasi-Newton method cannot be performed by a normal method because it is not differentiable at a connected portion of line segments. However, even in such a case, if sub-differentiation can be defined in a portion where differentiation cannot be performed as in piecewise linear approximation (sub-differentiation is possible), training using the quasi-Newton method can be performed. That is, the activation function used in the neuron cell element of the hidden unit such as the COS element only needs to be subdifferentiable.

  According to the above embodiment, generalization performance can be obtained by a smaller-scale calculation than in the past. Therefore, when the identification model is used in a mobile device with limited calculation performance, it is possible to provide a complete application and service in the mobile device.

[Realization by computer]
The character recognition device 180 and the character recognition NN learning device 186 according to the embodiment of the present invention can both be realized by computer hardware and a computer program executed on the computer hardware. FIG. 9 shows the external appearance of the computer system 330, and FIG. 10 shows the internal configuration of the computer system 330.

  Referring to FIG. 9, the computer system 330 includes a computer 340 having a memory port 352 and a DVD (Digital Versatile Disc) drive 350, a keyboard 346, a mouse 348, and a monitor 342.

  Referring to FIG. 10, in addition to the memory port 352 and the DVD drive 350, the computer 340 includes a CPU (Central Processing Unit) 356, a bus 366 connected to the CPU 356, the memory port 352, and the DVD drive 350, and a boot program. A read-only memory (ROM) 358 for storing etc., a random access memory (RAM) 360 connected to the bus 366 for storing program instructions, system programs, work data, etc., and a hard disk 354 are included. The computer system 330 further includes a network interface (I / F) 344 that provides a connection to a network 368 that allows communication with other terminals.

  A computer program for causing the computer system 330 to function as each function unit of the character recognition device 180 and the character recognition NN learning device 186 according to the above-described embodiment is a DVD 362 or a removable memory mounted on the DVD drive 350 or the memory port 352. 364 and further transferred to the hard disk 354. Alternatively, the program may be transmitted to the computer 340 through the network 368 and stored in the hard disk 354. The program is loaded into the RAM 360 when executed. The program may be loaded directly from the DVD 362 to the RAM 360 from the removable memory 364 or via the network 368.

  This program includes an instruction sequence including a plurality of instructions for causing the computer 340 to function as each function unit of the character recognition device 180 and the character recognition NN learning device 186 according to the above-described embodiment. Some of the basic functions necessary to cause the computer 340 to perform this operation are an operating system or a third party program running on the computer 340 or various dynamically linkable programming toolkits or programs installed on the computer 340 Provided by the library. For example, for the error back propagation method and the L-BFGS method, a software tool provided by a commercially available statistical processing library can be used. Therefore, this program itself does not necessarily include all functions necessary for realizing the system and method of this embodiment. This program is a system described above by dynamically calling an appropriate program in an appropriate function or programming toolkit or program library in execution in a controlled manner to obtain a desired result. It is only necessary to include an instruction for realizing a function as a device. Of course, all necessary functions may be provided only by the program.

[Application example]
The above embodiment relates to a handwritten character recognition device. However, the present invention is not limited to such an embodiment. For example, it can also be used for acoustic models and statistical language models used in speech recognition devices. It can also be applied to recognition of image patterns other than translation models and handwritten characters used in automatic translation devices.

  In the case of an acoustic model, instead of speech itself, as feature quantities, for example, MFCC feature quantities obtained from speech signals, LPC feature quantities, mel filter bank outputs, their primary and secondary differences, etc. Normally used feature values can be used as input. The number of nodes in the input layer is matched with the number of elements in the feature vector. The number of nodes in the output layer is matched with the assumed number of phonemes.

  In the case of a language model, for example: In the case of a trigram language model, consider a vector consisting of all possible vocabularies. A first vector in which the element corresponding to the word two words before is 1 and the other elements are 0, and a second vector in which the element corresponding to the word one word before is 1 and the other elements are 0 Are generated and connected to form one feature vector. The number of nodes in the input layer is therefore twice the number of vocabularies assumed. The number of output nodes matches the number of vocabularies assumed.

  In addition, the present invention can be applied to all machine learning and prediction devices using NN.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

180 Character recognition device 182 Input image 184 Character recognition text 186 Character recognition NN learning device 208 Character recognition NN
210 Decoder

Claims (6)

An input layer having a plurality of input nodes;
A nonlinear function layer having an input connected to receive the output from the input layer,
The nonlinear function layer includes a plurality of hidden layers each connected to receive outputs from the plurality of input nodes of the input layer, each capable of approximating an arbitrary independent component;
Each of the plurality of hidden layers is
A plurality of neuronal elements each having an input connected with a weight to the plurality of input nodes, and using an underdifferentiable periodic function as an activation function;
A neural network including an output aggregating element connected to receive outputs of the plurality of nerve cell elements in the same hidden layer with weights and using a subdifferentiable function as an activation function; The neural network according to claim 1, wherein the periodic function is a cosine function, a sine function, or a combination thereof, or a cosine function or a sine function that is piecewise linearly approximated. The neural network according to claim 1, wherein the neural network is trained by using L2 regularization for a weight parameter on an input side of each of the plurality of hidden layers. The neural network according to claim 1, wherein the neural network is trained using a quasi-Newton method. And an output layer connected to receive the output of the output aggregation elements of the plurality of hidden layers,
5. The output layer according to claim 1, wherein the output layer includes a plurality of output nerve cell elements each connected to receive the output of the output aggregation element of the plurality of hidden layers with a weight. Neural network described in 1. A computer program for causing a computer to function as the neural network according to any one of claims 1 to 5.

Images