JP5130934B2 - Recognition system, information processing apparatus, design apparatus, and program - Google Patents

Description

  The present invention calculates a value representing a category corresponding to an input value from a given input value using a predetermined calculation model, and derives the calculation model of the recognition system for recognizing the category corresponding to the input value, The present invention relates to a recognition system for recognizing a category by using this calculation model, an information processing apparatus used for implementing this method, a design apparatus, a program, and the like.

  Conventionally, a method using a neural network or a support vector machine is known as a method for realizing a recognition action on a computer (see, for example, Patent Document 1 and Non-Patent Document 1).

  A neural network is a model of a function of a nerve cell by a mathematical expression. A nerve cell has a function of emitting a pulse when a potential applied by an input signal exceeds a threshold value. In the neural network, such a function is realized by using a non-linear function such as a sigmoid function. That is, in the neural network, an operation is performed in which an input value is substituted into a nonlinear function and an output value is substituted into a nonlinear function corresponding to the next nerve cell. Then, an output value corresponding to the recognition result is obtained from the output value of the terminal nonlinear function.

  In addition, each synapse which connects between nerve cells has different propagation efficiency, and the result of recognition changes with the connection relation between nerve cells, and the propagation efficiency between each nerve cell. In the neural network, this is modeled by weighting the output value of the nonlinear function with the coupling load W and substituting it into the next nonlinear function.

  In constructing a recognition system using such a neural network, first, the type of neural network, that is, the connection relationship between units corresponding to nerve cells is determined. In other words, the type of the neural network is a neural network whose connection weight W is unknown. Therefore, next, the coupling load W is determined.

  Specifically, the combination load W is learned from a plurality of pieces of learning data by using, as learning data, a vector sample input to the neural network and a combination of output values to be obtained when the sample is input. As the neural network, a hierarchical network is known, and a back propagation method is known as a learning method of the connection weight W in the hierarchical network. Conventionally, a calculation model of a recognition system is derived by such a method, and the recognition system is constructed.

In addition, the recognition method by the support vector machine is to recognize the category corresponding to the vector U by calculating the value y representing the category corresponding to the input vector U using the discrimination function y ₁ or the discrimination function y ₂ .

尚、本願明細書では、ベクトルＡとベクトルＢとの内積を＜Ａ，Ｂ＞で表す。また、関数ｓｇｎ（ｘ）は、ｘ＞０のとき１をとり、ｘ≦０のとき−１をとる符号関数である。

In the present specification, the inner product of the vector A and the vector B is represented by <A, B>. The function sgn (x) is a sign function that takes 1 when x> 0 and takes −1 when x ≦ 0.

In addition, the parameter α _s (where s = 1, 2,..., S) and the parameter β included in the discrimination functions y ₁ and y ₂ are included in the sample U (s) of the vector U and the sample. Based on S pieces of learning data D _v (s) = {U (s), y (s)} (s = 1, 2,..., S) composed of combinations of values y (s) representing corresponding categories. The value is to be determined.

The function K (U, U (s), γ) is called a kernel, and the vectors φ (U), φ (when the vectors U, U (s) are nonlinearly transformed by the mapping φ and mapped to a high-dimensional space. U (s)) is used to calculate the inner product <φ (U), φ (U (s)>. The kernel K (U, U (s), γ) includes a Gaussian kernel K ₁ and a polynomial kernel K. ₂ is known.

識別関数ｙ₂は、非線形サポートベクタマシンと呼ばれ、この非線形サポートベクタマシンを用いた認識システムを構築するに際しては、まず、非線形サポートベクタマシンの型、即ち、パラメータα_s及びパラメータβ並びにパラメータγが未知の非線形サポートベクタマシンを決定する。この後、パラメータγを適当に設定して、学習データＤ_v（ｓ）＝｛Ｕ（ｓ），ｙ（ｓ）｝（ｓ＝１，２，…，Ｓ）に基づき、パラメータα_s及びパラメータβの解を求め、非線形サポートベクタマシンを設計する。

The discriminant function y ₂ is called a non-linear support vector machine. When constructing a recognition system using this non-linear support vector machine, first, the type of the non-linear support vector machine, that is, parameter α _s, parameter β and parameter γ are used. Determine an unknown nonlinear support vector machine. Thereafter, the parameter γ is appropriately set, and the parameter α _s and the parameter are set based on the learning data D _v (s) = {U (s), y (s)} (s = 1, 2,..., S). Find the solution of β and design a nonlinear support vector machine.

Incidentally, when the support vector machine design, as is well known to separate the different samples of categories, as from each group of samples having different categories, determining the hyperplane passing through the most distant position, the parameter alpha _s and parameters Find the solution of β.
JP 2005-316888 A Vladimir N. Vapnik, “The Nature of Statistical Learning Theory”, 2nd edition, (USA) Springer, 1999, p. 132-p. 140

However, the conventional neural network learning method and support vector machine design method have the following problems.
As described above, the back-propagation method is known as a conventionally known neural network learning method. However, in the back-propagation method, an output when a sample of input values indicated by learning data is input to the neural network is known. Since the connection load W is corrected in a direction to reduce the square error between the value and the output value (teacher signal) indicated by the learning data, an optimum solution is obtained depending on the initial value of the connection load W given during learning. There was a possibility that it was not possible.

  Here, the learning method of the connection weight W by the back propagation method will be described by taking a simple neural network as an example. Specifically, a three-layer feedforward neural network having two input units, one output unit, and two intermediate units, and a sigmoid function as a nonlinear function

が採用されたニューラルネットワーク（図１２（ａ）参照）を例に挙げて説明する。このニューラルネットワークの入出力関係は、次式で表される。

An example will be described in which a neural network (see FIG. 12A) is employed. The input / output relationship of this neural network is expressed by the following equation.

従って、このニューラルネットワークの結合荷重Ｗ＝｛ｗ１，…，ｗ９｝が、Ｓ個（ｓ＝１，２，…，Ｓ）の学習データＤ（ｓ）＝｛ｘ１（ｓ），ｘ２（ｓ），ｙ（ｓ）｝によって学習されるものとすると、バックプロパゲーション法では、二乗誤差Ｅ

Therefore, the connection load W = {w1,..., W9} of this neural network is S (s = 1, 2,..., S) learning data D (s) = {x1 (s), x2 (s). , Y (s)}, the backpropagation method uses the square error E

が最小となる結合荷重Ｗを求めることになる。尚、ｙ（ｓ）は、サンプルＸ（ｓ）＝｛ｘ１（ｓ），ｘ２（ｓ）｝に対応するカテゴリを表す値（ニューラルネットワークにて算出されるべき値）であり、例えば、−１又は＋１を採る。

The joint load W that minimizes the value is obtained. Note that y (s) is a value (a value to be calculated by a neural network) representing a category corresponding to the sample X (s) = {x1 (s), x2 (s)}. For example, −1 Or take +1.

  However, since the square error E is a non-linear function of w1,..., W9, as shown in FIG. 12 (b), there are a plurality of minimum values in this square error E, and w1,. Depending on the setting of the initial value of w9, learning may be performed so as to converge to a minimum point that is not the minimum value, and solutions of w1,..., w9 may be obtained. That is, in the conventional method, since only a local solution can be obtained for the coupling load W, there is a possibility that an appropriate solution for the coupling load W cannot be obtained.

  In addition, in the conventional method, the solution of the coupling load W is determined so that the square error E is reduced according to the learning data D (s). Therefore, the learning data D (s) other than the learning data D (s) used for calculating this solution is used. When the value of is input to the neural network, an appropriate recognition result is not always obtained.

  On the other hand, the support vector machine has the following problems. That is, to design a support vector machine, a set of samples φ (U (s)) when the samples U (s) are skipped to a higher dimension by the mapping φ can be linearly separated into sets for each category. If it is not possible to set an appropriate mapping φ such that the amount of computation does not become a huge amount as a mapping φ that enables linear separation of a set of φ (U (S)), There is a problem that a recognition system using a support vector machine cannot be designed as a recognition system corresponding to the sample U (s).

In order to simplify the calculation of the inner product <φ (U), φ (U (s))> in the discrimination function y ₂ , the above-described kernel K (U, U (s), γ) is used. Although, as is well known, an arbitrary function cannot be taken as the kernel K, the arbitrary learning data can be used even when a nonlinear support vector machine is designed using the kernel K (U, U (s), γ). There was a problem that it was not possible to design a non-linear support vector machine corresponding to.

  Further, the value of the parameter γ of the kernel K (U, U (s), γ) has conventionally been determined by the designer through trial and error, and this may hinder the design of the optimal nonlinear support vector machine. It was.

  The present invention has been made in view of these problems, and an object of the present invention is to be able to derive a calculation model of a recognition system that is more suitable than before and to construct a preferable recognition system. To do.

In order to achieve this object, the present invention provides a category corresponding to the input value X from the input value X = {x1,..., XN ₁ } (where the value N ₁ is an integer of 2 or more). Is calculated by a predetermined calculation model, and the calculation model of the recognition system for recognizing the category corresponding to the input value X represents the sample X (s) of the input value X and the category to which this sample belongs. Based on arbitrary S pieces of learning data D (s) = {X (s), y (s)} (where s = 1,..., S) composed of combinations of values y (s). This is a model derivation method for deriving a calculation model of a recognition system through the following procedure [a] to procedure [c].

In the model derivation method of the present invention, first, as a neural network to be learned, a (L ₁ +1) layer neural network in which the 0th layer is an input layer composed of N ₁ input units and the L ₁ layer is an output layer. A network (however, the value L ₁ is an integer of 2 or more) is set.

Then, in this neural network, an unknown number of learning parameters W ₁ are learned based on the S learning data D (s) using the value y (s) of each learning data D (s) as a teacher signal. A solution of W ₁ is obtained (procedure [a]). Incidentally, the solution of learning parameters W ₁ is, Ru can be obtained by known back propagation method.

In addition, after the solution of the learning parameter W ₁ is obtained in the procedure [a], each learning data is obtained using a (L ₁ +1) layer neural network after learning obtained by setting the solution of the learning parameter W _1. New learning data D _v (s) corresponding to D (s) is generated (procedure [b]).

Specifically, for each learning data D (s), a sample X (s) of the learning data D (s) is used as an input value X to the neural network of the (L ₁ +1) layer after learning. output value Z = {z1, ..., zN 2} of the (L ₁ -1) layer constituting the network seeking to set the output value Z obtained above as a new sample U (s), a new set Data D _v (s) = {U (s), y (s)} consisting of a combination of the sample U (s) and the value y (s) indicated by the learning data D (s) is used as the learning data D (s). ) _Is generated as new learning data D _v (s) corresponding to. However, the (L ₁ -1) th layer is composed of N ₂ intermediate units (where the value N ₂ is an integer of 2 or more).

Further, after the completion of the procedure [b], a value y representing a category corresponding to the input value U = {u1,..., UN} is calculated based on each learning data D _v (s) generated in the procedure [b]. A non-linear support vector machine is designed (procedure [c]). However, the value N is the number of dimensions of the sample U (s) of the learning data D _v (s) that is a sample of the input value U.

  The non-linear support vector machine is a discriminant function y

により、入力値Ｕに対応するカテゴリを表す値ｙとして、＋１又は−１を算出するものである。

Thus, +1 or −1 is calculated as the value y representing the category corresponding to the input value U.

Therefore, here, after setting the value of the parameter γ in the nonlinear support vector machine type, that is, the nonlinear support vector machine in which the parameters α _s, β, and γ are unknown, the learning data D _v (s) = { Based on U (s), y (s)} (s = 1, 2,..., S), a solution of the parameter α _s and the parameter β is obtained, and this solution is set in the nonlinear support vector machine to obtain the nonlinear support vector. Design the machine.

In the present invention, the following calculation model is derived as the calculation model of the recognition system using the learning parameter W ₁ obtained by these steps [a] to [c] and the nonlinear support vector machine.

That is, a combination of the calculation model capable of calculating the output value Z of the (L ₁ -1) layer in the (L ₁ +1) layer neural network after learning and the nonlinear support vector machine designed in step [c]. An output value Z of the (L ₁ -1) -th layer is calculated from the input value X, and this output value Z is used as the input value U of the nonlinear support vector machine, and the output value of the nonlinear support vector machine A calculation model for calculating y is derived as a calculation model of the recognition system.

In the present invention, the calculation model of the recognition system is derived in such a way that the recognition method using the neural network has a problem that it is not theoretically guaranteed that the optimum solution of the learning parameter W ₁ can be obtained. This is because the recognition method using the nonlinear support vector machine that can obtain the optimal solution of α _s and parameter β has a problem that the nonlinear support vector machine cannot be designed for an arbitrary recognition target.

  Compared to neural networks, nonlinear support vector machines are superior in that they can obtain correct recognition results for inputs other than learning data. For example, the sample U (s) of the input value U needs to be a set that is easily linearly separated, and the conventional method cannot adopt the nonlinear support vector machine method for an arbitrary recognition system. There was a problem.

Therefore, in the present invention, new learning data D _v (s) is generated from the learning data D (s), and the sample X (s) is replaced with a value Z that is easily linearly separated, thereby obtaining the sample X (s). Regardless of this, a nonlinear support vector machine can be designed.

Incidentally, the value Z is easily linearly separable, the output y of the output unit in the neural network, the output value of it than the one lower layer first (L ₁ -1) layer Z = {z1, ..., zN 2} of This is because it represents a value corresponding to the signed distance from the hyperplane.

That is, the output y of the output unit in the neural network is expressed by W (i) representing the coupling coefficient between the i-th intermediate unit and the output unit in the (L ₁ -1) layer, and the threshold value of the output unit represented by W0. ,

で表すことができる。

Can be expressed as

On the other hand, the signed distance L _g from the hyperplane in the N ₂ dimension is as follows, assuming that the normal vector of the hyperplane is a normal vector G = {g1,..., GN ₂ } of length 1.

で表すことができる。

Can be expressed as

Therefore, if an appropriate solution of the learning parameter W (i) corresponding to the learning data D (s) is obtained, the (L ₁ −1) layer when each sample X (s) is input to the neural network. The set of output values Z can be linearly separated on the hyperplane for each category.

Of course, the type of the neural network is set by the trial and error of the designer, and when the type of the neural network adopted by the designer, in other words, when the number of layers of the neural network or the number of intermediate units is not appropriate, There is a possibility that the set of output values Z of the (L ₁ -1) layer cannot be linearly separated in the hyperplane. However, it can be said that at least the set of output values Z is a set that is more easily linearly separated than the set of samples X (s).

Therefore, in the present invention, new learning data D _v is generated based on the output value Z, and a nonlinear support vector machine is designed based on the learning data D _v . Since the set of output values Z is a set that is much easier to linearly separate than at least X (s), linear separation can be performed relatively easily by the mapping φ.

  Therefore, according to the present invention, an appropriate nonlinear support vector machine can be designed regardless of the distribution of the sample X (s), and a calculation model of a recognition system that is more suitable than the conventional model can be derived using the nonlinear support vector machine. It can be done.

In other words, since only a local solution can be obtained as a solution of the learning parameter W ₁ (coupling coefficient) in the neural network, there is no guarantee that a theoretically optimal neural network can be designed. However, in the present invention, the parameter α _s Since the calculation model of the recognition system is derived using a support vector machine that can obtain a global solution as a solution of the parameter β, the above-mentioned drawbacks of the neural network can be complemented by the support vector machine. An appropriate calculation model can be obtained as a calculation model of the system.

  In addition, in a neural network, when a value other than learning data is input, an appropriate recognition result is not always obtained. However, in a support vector machine, as described above, values other than learning data are also correct. Since a recognition result can be obtained, it is possible to construct a recognition system that has a better recognition performance than the prior art by using a calculation model incorporating a support vector machine derived by the method of the present invention.

That is, the value y representing the category corresponding to the input value X is calculated from the given input value X = {x1,..., XN ₁ } by the calculation model derived by the model derivation method of the present invention, and the input value if build recognizing system the category corresponding to the X, in the recognition system, it is possible to achieve excellent recognition acts than before.

The recognition system described above outputs a value y representing a category corresponding to the input value X as a recognition result of the category corresponding to the input value X, and the nonlinear support vector obtained when calculating the value y. the absolute value of the input value p to sign function constituting the machine | p | a, have good when it is in the configuration of outputting the information indicating the accuracy of the recognition result. If the recognition system is configured in this way, it is possible to output information related to the accuracy of the recognition result. Therefore, when the accuracy is low, an action such as re-performing the recognition operation becomes possible.

In the model derivation method described above, the learning data D _v (s) may be generated as follows. That is, in step [b], for each learning data D (s), the value Z calculated based on the sample X (s) of the learning data D (s) is reduced in dimension by a predetermined algorithm, and the dimension reduction is performed. The subsequent value V = {v1,..., VN ₃ } (where the value N ₃ is an integer greater than or equal to ₂ smaller than the value N ₂ ) is set as a new sample U (s), and the learning data D _v (s) = {U (s), y (s)} may be generated.

When the learning data D _v (s) is generated in this way, the output value Z of the (L ₁ −1) -th layer is calculated from the input value X as a calculation model of the recognition system, and this output value Z is calculated as above. The dimension is reduced by a predetermined algorithm, and the output value Z is converted into an N ₃ dimensional value V = {v1,..., VN ₃ }, and the converted value V is used as the input value U of the nonlinear support vector machine. a calculation model for calculating the output value y of the support vector machine, ing to be derived. Thus, if the input value U to the nonlinear support vector machine is reduced in dimension, the amount of calculation related to the design and recognition action of the nonlinear support vector machine can be suppressed.

Specifically, the dimension reduction can be realized by a principal component analysis technique. In other words, in step [b], for each learning data D (s), N ₃ values Z calculated based on the sample X (s) of the learning data D (s) are obtained by the principal component analysis technique. N ₂ dimensional vector Jm (where m = 1,..., N ₃ ) is used to reduce the dimension, and the inner product of the value J of the m-th element is the vector Jm and the value Z. <Jm, Z> value V = {v1, ..., vN _3} of N _{3-dimensional} represented by converting the value V = after the dimension reduction {v1, ..., vN _3} a new sample U (S) may be set to generate learning data D _v (s) = {U (s), y (s)}.

When the learning data D _v (s) is generated in this way, the output value Z of the (L ₁ −1) -th layer is calculated from the input value X as a calculation model of the recognition system, and this output value Z is Using N ₃ N _two- dimensional vectors Jm, the dimension is reduced, and a calculation model for calculating the output value y of the nonlinear support vector machine is derived using the value V after the dimension reduction as the input value U of the nonlinear support vector machine. in particular ing.

When the kernel K (U, U (s), γ) is used for the nonlinear support vector machine, the parameter γ is set before obtaining the solution of the parameter α _s and the parameter β based on the learning data D _v. Although it is necessary, designing the parameter γ using the conventional method is not only troublesome, but it is not always possible to set an appropriate value for the parameter γ. Therefore, a solution for the parameter γ is obtained as follows. Good.

  That is, in the procedure [c], for the nonlinear support vector machine that calculates the value y representing the category from the input value U, the parameter γ of the kernel K (U, U (s), γ) that constitutes the nonlinear support vector machine. Is determined by the following steps [1] to [3], and a non-linear support vector machine is designed using the kernel K (U, U (s), γ) in which the appropriate value is set as the parameter γ. Good.

Procedure [1]: a learning object of a neural network, the layer 0 as the input layer of N input unit, the neural network of the first L ₂ layer was output layer (L ₂ +1) layer (where the value L ₂ is an integer greater than or equal to 2.) is set. Then, the learning parameters W ₂ unknowns in the neural network, using the value y (s) of the training data D _v (s) as a teacher signal, and learning based on the S learning data D _v (s), A solution for the learning parameter W ₂ is obtained.

Procedure [2]: learning each learning data D _v (s), comprising the training data D _v samples U (s) of (s), by setting the solution to the learning parameters W ₂ determined in Step [1] As an input value to the later (L ₂ +1) layer neural network, the output value H (s) = {h 1 (s),..., HN _{h of} the (L ₂ −1) layer constituting this neural network. (S)} is obtained. However, the (L ₂ −1) -th layer is composed of N _h intermediate units (where the value N _h is an integer larger than the dimension number N of the input value U).

Procedure [3]: Using the learning data D _v (s) and the value H (s) corresponding to the sample U (s) of each learning data D _v (s) obtained in procedure [2],

に従い、二乗誤差Ｅが極小値を採るパラメータγの解γ^*を、パラメータγの適値として求める。

Accordingly, a solution γ ^* of the parameter γ for which the square error E takes a minimum value is obtained as an appropriate value of the parameter γ.

If the solution of the parameter γ is obtained by such procedure [1] to procedure [3], an appropriate value of the parameter γ to be set in the kernel can be easily obtained by calculation, and the parameter γ can be obtained by trial and error as in the past. than when setting the value, the appropriate value, Ru can be set to the parameter gamma.

As the kernel K (U, U (s), γ), a Gaussian kernel or a polynomial kernel can be used. When using the Gaussian kernel, Q′ab is expressed by the following equation in step [3]. ing to be determined in.

また、多項式カーネルを用いる場合には、手順［３］において、Ｑ’ａｂを、次式で求めることができる。

Further, when using a polynomial kernel, Q′ab can be obtained by the following equation in the procedure [3].

また、本発明の方法で、認識システムの計算モデルを導出するに際しては、次の情報処理装置を用いるとよい。

Further, in the method of the present invention, when deriving the computational model of the recognition system, it has good With next information processing apparatus.

The information processing apparatus according to the present invention calculates a value y representing a category corresponding to the input value X from a given input value X = {x1,..., XN ₁ } by using a predetermined calculation model. An information processing apparatus used for derivation of the calculation model of the recognition system for recognizing a category corresponding to the following, comprising the following acquisition means, learning means, new data generation means, support vector machine design means, and output means Is.

In this information processing apparatus, the acquisition means includes S pieces of learning data D (s) = {X ( s), y (s)}, and the learning means obtains a solution of the learning parameter W ₁ of the neural network by the above-described procedure [a] based on each learning data D (s) obtained by the obtaining means. Ask.

Further, the new data generation means is obtained by using the (L ₁ +1) layer neural network after learning, in which the solution of the learning parameter W ₁ obtained by the learning means is set by the procedure [b] described above. New learning data D _v (s) corresponding to each learning data D (s) acquired by the means is generated, and the support vector machine designing means, based on each learning data D _v (s), the above procedure [ c] to design a non-linear support vector machine.

In this information processing apparatus, the output means outputs information representing the solution of the learning parameter W ₁ obtained by the learning means and information representing the nonlinear support vector machine designed by the support vector machine design means. . The information representing the non-linear support vector machine may include information such as the parameter α _s and the parameter β constituting the discriminant function y as the non-linear support vector machine, the kernel parameter γ, and the like.

  If the information processing apparatus configured in this way is used, the user can easily provide the learning data D (s) to the information processing apparatus, and the calculation model of the recognition system corresponding to the learning data D (s) can be easily obtained. The information concerning can be obtained.

Note that the value Z calculated based on the sample X (s) of the learning data D (s) is dimension-reduced by the new data generating means, and the value V = {v1,..., VN ₃ } after the dimension reduction is obtained. When the learning data D _v (s) = {U (s), y (s)} is set as a new sample U (s), the output unit learns the learning parameter W ₁ obtained by the learning unit. information representing the solution of, and information indicating the non-linear support vector machine designed by the support vector machine design means, and, Ru can be configured to output the information from the value Z representing the method of converting the value V .

By using the information processing apparatus configured as described above, it is possible to derive a calculation model of a recognition system that can realize a recognition action with a small amount of calculation.
When the dimension Z of the value Z calculated based on the sample X (s) of the learning data D (s) is reduced using N ₃ N _two- dimensional vectors Jm obtained by the principal component analysis technique, output means as information indicating the conversion method from the value Z to a value V, Ru can be configured to output the N ₃ pieces of N _{2-dimensional} vector Jm.

Moreover, the acquisition unit of the information processing apparatus of the present invention is provided, the learning means, new data generation means, support vector machine design means, and, the function as the output unit, the program, Ru can be realized on the computer.

In addition, design equipment of the present invention, the input value U = {u1, ..., uN } given from an apparatus for designing a nonlinear support vector machines for calculating a value y representing the category of the input value U S pieces of learning data D _v (s) = {U (s), y (s)} consisting of a combination of a sample U (s) and a value y (s) representing a category to which the sample belongs (where s = 1 ,..., S)) and a kernel K (U, U (s), which is employed in the nonlinear support vector machine based on the S learning data D _v (s) acquired by the acquisition unit. An appropriate value calculating means for obtaining an appropriate value of the parameter γ of γ) by the above-mentioned procedure [1] to procedure [3], and a kernel K (U, U) in which the appropriate value of the parameter γ obtained by the appropriate value calculating means is set. S learning data D _v acquired by the acquiring means using (s), γ) On the basis of (s), a design means for designing a non-linear support vector machine and an output means for outputting information representing the non-linear support vector machine designed by the design means are provided.

With this design apparatus, the user, to the extent that gives the training data D _v (s) in the design system, it is possible to obtain information of the preferred nonlinear support vector machine learning data D _v (s), effectively With the method of the present invention, a calculation model of the recognition system can be derived.

The design device of the present invention, the kernel K as a parameter (U, U (s), γ) γ, Ru can be configured to solving parameter gamma Gaussian kernel. Moreover, acquiring means for designing device comprises of the present invention, suitable value calculating means, the design unit, and the function as the output unit, the program, Ru can be realized on the computer.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of a model deriving device 1 of the first embodiment to which the present invention is applied. As shown in FIG. 1, the model deriving device 1 according to the present embodiment is used as a work area when executing a program, a CPU 11 that performs various arithmetic processes, a ROM 13 that stores a boot program and the like, as in a known personal computer. RAM 15, hard disk device 17 that stores an operating system and other various programs and data, a display device 21 that includes a liquid crystal display, a user interface 23 that includes a keyboard, a pointing device, and the like, and data read / write from / to a flexible disk And a possible drive device 25.

  The model deriving device 1 operates by an operating system recorded on the hard disk device 17. For example, when a program execution command is input through the user interface 23, the model deriving device 1 performs processing based on the program specified by the user. It is executed by the CPU 11 under the management of the system.

  Specifically, the model deriving device 1 includes a model deriving program for deriving a calculation model of the recognition system in the hard disk device 17. FIG. 2 is a flowchart showing a model derivation process executed by the CPU 11 according to the model derivation program.

  When the execution instruction of the model derivation program is input through the user interface 23, the CPU 11 executes the processing shown in FIG. 2, and the calculation model of the recognition system is a combination of a neural network and a non-linear support vector machine. The design value is output to the data file and output to the display device 21.

  FIG. 3A is a basic configuration diagram of a calculation model derived by this model derivation process. In this embodiment, as a calculation model of a recognition system for calculating a value y representing a category from input values X = {x1,..., XN1} (where N1 is an integer value of 2 or more), a neural network is used in the first half. In the second half, a calculation model for calculating a value y representing a category is derived by performing calculations using a nonlinear support vector machine. FIG. 3B is a diagram showing a configuration of a recognition system configured by a conventional neural network as a comparison with the calculation model derived in the present embodiment, and FIG. It is a figure showing the structure of the recognition system comprised with the nonlinear support vector machine.

  When the model derivation process shown in FIG. 2 is started, the CPU 11 first displays a file selection screen having a GUI configuration on the display device 21 in S110, and data used for parameter learning of the calculation model to the user through the file selection screen. Is selected as a data file to be read. Specifically, in S110, a list of data files recorded in the hard disk device 17 and the drive device 25 and corresponding to the model derivation program is displayed on the GUI configuration file selection screen.

  Then, when a data file to be read is selected through the file selection screen (Yes in S120), the selected data file is read, and a three-layer feedforward neural network that learns in subsequent processing according to the description content of the data file The number N1 of network input units is set (S130), and the number N2 of intermediate units of the neural network is set (S140). However, N1 and N2 to be set are integer values of 2 or more.

  When the processing of S140 is finished, the neural network is different from the neural network according to the description content of the read data file, and the neural network used for deriving the kernel parameter solution of the nonlinear support vector machine The number Nh of intermediate units is set (S150), and then the total number S of learning data is set (S160). However, Nh to be set is an integer value of 2 or more larger than N2. When this process ends, the process proceeds to S170.

  FIG. 4A shows the structure of the data file read out in the model derivation process. As shown in FIG. 4A, in the data file read by the model derivation process, user set values for the number of input units N1, the number of intermediate units N2, Nh, and the total number S of learning data are described. In S130 to S160, the values of the parameters N1, N2, Nh, and S are set according to the description content.

  Further, in the data file, as learning data, S sets of samples of the input value X = {x1, x2,..., XN1} and a value y (+1 or −1) representing the category to which the sample belongs are described. In S170 to S200, S pieces of learning data D (1), D (2),..., D (S) used for the subsequent processing are set according to the description content.

  That is, the parameter s is set to the value 1 in S170, and the parameter X (s) = {x1 (s),... Constituting the data D (s) = {X (s), y (s)} in S180. , XN1 (s)} is set to the value of the sample X = {x1, x2,..., XN1} described in the sth from the top in the data file, and the data D (s) = {X (s), A value y representing the category to which the sample described in the data file belongs is set in the parameter y (s) constituting y (s)}. In S190, the value of the parameter s is incremented by 1, and in S200, it is determined whether or not the value of the parameter s after the addition is larger than the total number S of samples. If s ≦ S (No in S200), the process proceeds to S180, and if s> S, the process proceeds to S210. In this way, in S170 to S200, S pieces of learning data D (1), D (2),..., D (S) are set.

In S210, the CPU 11 determines a three-layer feedforward neural network of N1 input units, N2 intermediate units, and 1 output unit based on S pieces of learning data D (1),..., D (S). Unknown learning parameter W1 = {Wa (0,1), ..., Wa (i, j), ..., Wa (N1, N2), Wb (0), ..., Wb (j), ..., Wb (N2 )} the solution W1 ^* of the value y (s) of the training data D (s) as a teacher signal, determined by the well-known back propagation method.

  As shown in FIG. 5A, the parameter Wa (i, j) is a parameter corresponding to the coupling coefficient between the i-th input unit and the j-th intermediate unit, and the parameter Wb (j) is This is a parameter corresponding to the coupling coefficient between the intermediate unit and the output unit. The parameter Wa (0, j) is a parameter corresponding to the threshold value of the j-th intermediate unit, and the parameter Wb (0) is a parameter corresponding to the threshold value of the output unit.

That is, in S210, as a solution W1 ^* learning parameters W1, (N1 · N2 + N2 ) number of learning parameters Wa (i, j) (where, i = 0,1, ..., N1 , j = 1,2, ..., Solution Wa ^* (i, j) of (N2) and solution Wb ^* (j of (N2 + 1) learning parameters Wb (j) (where j = 0, 1,..., N2). )

  The output value y of the three-layer feedforward neural network having N1 input units, N2 intermediate units, and one output unit with input values X = {x1,..., XN1} can be expressed by the following equation. However, x0 = 1 and z0 = 1. The function f (x) is a non-linear function, and in this embodiment, a sigmoid function is adopted (f (x) = sig (x)).

従って、Ｓ２１０では、上式に基づき、サンプルＸ（ｓ）が属するカテゴリを表す値ｙ（ｓ）と、当該サンプルＸ（ｓ）をニューラルネットワークに入力したときの出力値ｙ（Ｗ１，Ｘ（ｓ））との二乗誤差Ｅが最小となる値Ｗ１^*を、勾配法により、学習パラメータＷ１の解Ｗ１^*として求める。

Therefore, in S210, based on the above equation, the value y (s) representing the category to which the sample X (s) belongs, and the output value y (W1, X (s) when the sample X (s) is input to the neural network. The value W1 ^* that minimizes the square error E)) is determined as the solution W1 ^* of the learning parameter W1 by the gradient method.

  That is, the square error E is expressed by the following equation:

で定義すると共に、学習パラメータＷａ（ｉ，ｊ），Ｗｂ（ｊ）の修正量ΔＷａ（ｉ，ｊ），ΔＷｂ（ｊ）を、学習速度を表す定数ηを用いて次式

And the correction amounts ΔWa (i, j) and ΔWb (j) of the learning parameters Wa (i, j) and Wb (j) are expressed by the following equation using a constant η representing the learning speed:

で定め、学習パラメータＷａ（ｉ，ｊ），Ｗｂ（ｊ）の値を、初期値からΔＷａ（ｉ，ｊ），ΔＷｂ（ｊ）ずつ更新していく。そして、学習パラメータＷａ（ｉ，ｊ），Ｗｂ（ｊ）の値が収束したときの当該学習パラメータＷａ（ｉ，ｊ），Ｗｂ（ｊ）の値を、二乗誤差Ｅが最小となる値Ｗ１^*として求める。尚、ここでは「二乗誤差Ｅが最小」と表現しているが、上述の手法では、二乗誤差Ｅが局所的に最小となる値Ｗ１^*しか求めることができない。即ち、Ｓ２１０では、実質的には、二乗誤差Ｅが極小値となる値Ｗ１^*を、学習パラメータＷ１の解Ｗ１^*として求めることになる。

The learning parameters Wa (i, j) and Wb (j) are updated from the initial values by ΔWa (i, j) and ΔWb (j). Then, when the values of the learning parameters Wa (i, j) and Wb (j) converge, the values of the learning parameters Wa (i, j) and Wb (j) are set to a value W1 ^* that minimizes the square error E. Asking. Although the expression “the square error E is minimum” is used here, only the value W1 ^* at which the square error E is locally minimum can be obtained by the above-described method. In other words, in S210, the value W1 ^{* at} which the square error E is minimal is obtained as the solution W1 ^* of the learning parameter W1.

  When the process in S210 is completed, the CPU 11 proceeds to S220 and executes a new learning data generation process shown in FIG. FIG. 6 is a flowchart showing new learning data generation processing executed by the CPU 11.

  When the new learning data generation process is started in S220, the CPU 11 first sets the parameter N to the value N2, and generates an N = N2-dimensional parameter U (s) (s = 1,..., S) ( S221). The parameter s is set to a value 1 (S222).

Thereafter, the process proceeds to S223, and the output value Z = {of the intermediate layer when the sample X (s) is input to the three-layer feedforward neural network in which the solution W1 ^* of the learning parameter W1 calculated in S210 is set. z1,..., zN2} are obtained based on the learning data D (s).

  Specifically, the output value Z of the intermediate layer is calculated according to the following equation (provided that x0 (s) = 1).

また、この処理後には、算出した値Ｚを、パラメータＵ（ｓ）＝｛ｕ１（ｓ），…，ｕＮ（ｓ）｝に設定し、新たな学習データＤｖ（ｓ）＝｛Ｕ（ｓ）＝Ｚ，ｙ（ｓ）｝を生成する（Ｓ２２５）。その後、パラメータｓを１加算した値に更新し（Ｓ２２７）、更新後のパラメータｓの値が学習データの総数Ｓよりも大きいか否かを判断する（Ｓ２２９）。

After this processing, the calculated value Z is set to the parameter U (s) = {u1 (s),..., UN (s)}, and new learning data Dv (s) = {U (s) = Z, y (s)} is generated (S225). Thereafter, the parameter s is updated to a value obtained by adding 1 (S227), and it is determined whether or not the updated parameter s value is larger than the total number S of learning data (S229).

  If s ≦ S (No in S229), the process proceeds to S223. In this way, by repeatedly executing the processing of S223 to S229, in the new learning data generation processing, from the learning data D (s) to the learning data Dv (s) in the range of s = 1 to s = S. ) Is generated. When generation of new learning data Dv (s) is completed for all learning data D (s), it is determined Yes in S229, and the new learning data generation processing ends.

  In S220, when the new learning data generation process ends, the CPU 11 proceeds to S230 and executes the kernel parameter setting process shown in FIG. FIG. 7 is a flowchart showing kernel parameter setting processing executed by the CPU 11.

When the kernel parameter setting process is started, the CPU 11 sets a three-layer feedforward neural network having N input units, Nh intermediate units, and one output unit as a new learning target neural network. learning parameters W2 = {Wc (0,1), ..., Wc (j, k), ..., Wc (N, Nh), Wd (0), ..., Wd (Nh)} solutions W2 ^* of at S220 Based on the generated S pieces of learning data Dv (1), Dv (2),..., Dv (S), the value y (s) indicated by the learning data Dv (s) is used as a teacher signal by the back propagation method. (S231).

  As shown in FIG. 5B, the parameter Wc (j, k) is a parameter corresponding to the coupling coefficient between the jth input unit and the kth intermediate unit, and the parameter Wd (k) is the kth This is a parameter corresponding to the coupling coefficient between the intermediate unit and the output unit. The parameter Wc (0, k) is a parameter corresponding to the threshold value of the k-th intermediate unit, and the parameter Wd (0) is a parameter corresponding to the threshold value of the output unit.

That is, in S231, as the solution W2 ^* of learning parameters W2, (N · Nh + Nh ) number of learning parameters Wc (j, k) (however, j = 0,1, ..., N , k = 1,2, ..., Nh.) Solution Wc ^* (j, k) and solution Wd ^* (k) of (Nh + 1) learning parameters Wd (k) (where k = 0, 1,..., Nh). Ask for.

  The output value y of the three-layer feedforward neural network with N input units, Nh intermediate units, and one output unit with input values U = {u1,..., UN} can be expressed as , U0 = 1 and h0 = 1.)

従って、Ｓ２３１では、上式に基づき、サンプルＵ（ｓ）が属するカテゴリを表す値ｙ（ｓ）と、当該サンプルＵ（ｓ）をニューラルネットワークに入力したときの出力値ｙ（Ｗ２，Ｕ（ｓ））との二乗誤差Ｅが最小となる値Ｗ２^*を、勾配法により、学習パラメータＷ２の解Ｗ２^*として求める。

Therefore, in S231, based on the above equation, the value y (s) representing the category to which the sample U (s) belongs, and the output value y (W2, U (s) when the sample U (s) is input to the neural network. A value W2 ^* that minimizes the square error E)) is obtained as a solution W2 ^* of the learning parameter W2 by the gradient method.

但し、Ｓ２３１では、Ｓ２１０と同様、実質的には、二乗誤差Ｅが極小値となる値Ｗ２^*を、学習パラメータＷ２の解Ｗ２^*として求めることになる。

However, in S231, as in S210, the value W2 ^{* at} which the square error E is minimal is obtained as the solution W2 ^* of the learning parameter W2.

When the process in S231 is completed, the CPU 11 sets the parameter s to a value 1 (S232), and then proceeds to S233.
When the process proceeds to S233, the CPU 11 outputs the output value H of the intermediate layer when the sample U (s) is input to the three-layer feedforward neural network in which the solution W2 ^* of the learning parameter W2 calculated in S231 is set. (S) = {h1 (s),..., HNh (s)} is obtained based on the learning data Dv (s).

  Specifically, the output value H of the intermediate layer is calculated according to the following equation (provided that u0 (s) = 1).

また、この後には、パラメータｓの値を１加算し（Ｓ２３４）、Ｓ２３５に移行する。そして、Ｓ２３５では、更新後のパラメータｓの値が学習データの総数Ｓよりも大きいか否かを判断し、ｓ≦Ｓである場合には（Ｓ２３５でＮｏ）、Ｓ２３３に移行する。このようにして、当該カーネルパラメータ設定処理では、Ｓ２３３〜Ｓ２３５の処理を繰返し実行することにより、ｓ＝１からｓ＝Ｓまでの範囲において、学習データＤｖ（ｓ）のサンプルＵ（ｓ）を上記ニューラルネットワークに入力したときに得られる中間層の出力値Ｈ（ｓ）を算出する。そして、ｓ＞Ｓとなると、Ｓ２３５でＹｅｓと判断し、Ｓ２３６に移行する。

Thereafter, the value of the parameter s is incremented by 1 (S234), and the process proceeds to S235. In S235, it is determined whether or not the value of the updated parameter s is larger than the total number S of learning data. If s ≦ S (No in S235), the process proceeds to S233. In this way, in the kernel parameter setting process, by repeatedly executing the processes of S233 to S235, the sample U (s) of the learning data Dv (s) is stored in the range from s = 1 to s = S. An intermediate layer output value H (s) obtained when input to the neural network is calculated. When s> S, it is determined Yes in S235, and the process proceeds to S236.

In S236, the CPU 11 determines that the element Q _ab in the a-th row and the b-th column is an inner product <H (a), H (b)> of H (s = a) and H (s = b). A matrix Q of S rows and S columns represented is calculated.

また、この処理を終えると、ＣＰＵ１１は、Ｓ個の学習データＤｖ（１），…，Ｄｖ（Ｓ）に基づき、次の二乗誤差Ｅが最小となるパラメータγの解γ^*を、勾配法により算出する（Ｓ２３７）。

When this processing is completed, the CPU 11 calculates a solution γ ^{* of the} parameter γ that minimizes the next square error E based on the S pieces of learning data Dv (1),..., Dv (S) by the gradient method. Calculate (S237).

尚、本実施例では、カーネルとしてガウシアンカーネルが採用されているものとする。また、Ｓ２３７では、勾配法によりパラメータγの解γ^*を求めるため、実質的には、二乗誤差Ｅが極小値となる値γ^*を、パラメータγの解γ^*として求めることになる。

In this embodiment, a Gaussian kernel is adopted as the kernel. In S237, since the solution γ ^* of the parameter γ is obtained by the gradient method, the value γ ^{* at} which the square error E becomes the minimum value is substantially obtained as the solution γ ^* of the parameter γ.

When the processing in S237 is completed, the CPU 11 sets the calculated solution γ ^* as the parameter γ of the kernel K (U, U (s), γ), and sets the kernel K used for the design of the nonlinear support vector machine. The following settings are made (S238).

その後、当該カーネルパラメータ設定処理を終了する。

Thereafter, the kernel parameter setting process ends.

When the kernel parameter setting process in S230 ends, the CPU 11 proceeds to S240 and designs a nonlinear support vector machine.
Specifically, in S240, the solutions α _s ^* and β ^* of the unknown parameters α _s and β in the nonlinear support vector machine are obtained by solving the following constrained secondary optimization problem as in the known technique. .

  That is, restraint conditions

の下で、目的関数Ｌα

The objective function Lα

が最大となるパラメータα_s（ｓ＝１，２，…，Ｓ）の解α_s ^*を、非線形サポートベクタマシンのパラメータα_sの解α_s ^*として算出する。

The solution α _s ^* of the parameter α _s (s = 1, 2,..., S) that maximizes is calculated as the solution α _s ^* of the parameter α _s of the nonlinear support vector machine.

Further, a solution β ^{* of the} parameter β is obtained by the following equation using U (s) corresponding to α _s ^* (s = 1, 2,..., S) that is not zero, that is, a support vector.

このようにして、Ｓ２４０で非線形サポートベクタマシンのパラメータα_s，βの解α_s ^*，β^*を算出すると、ＣＰＵ１１は、Ｓ２５０に移行し、上述のＳ２１０で算出した学習パラメータＷａ（ｉ，ｊ）の解Ｗａ^*（ｉ，ｊ）、即ち、ニューラルネットワークにおける中間層の出力値Ｚの算出に必要な学習パラメータＷａ（ｉ，ｊ）の解Ｗａ^*（ｉ，ｊ）、及び、Ｓ２４０で設計した非線形サポートベクタマシンのパラメータα_s，β，γの解α_s ^*，β^*，γ^*を記述したデータファイルを生成し、これを、ハードディスク装置１７に書き込む。尚、図４（ｂ）は、Ｓ２５０で出力するデータファイルの構成を示した説明図である。

In this way, when the solutions α _s ^* and β ^* of the parameters α _s and β of the nonlinear support vector machine are calculated in S240, the CPU 11 proceeds to S250, and the learning parameter Wa (i, j calculated in S210 described above is obtained. solution Wa ^* (i, j)) of, i.e., solutions of learning parameters required for calculating the output value Z of the intermediate layer Wa (i, j) in the neural network Wa ^* (i, j), and, designed in S240 A data file describing the solutions α _s ^* , β ^* , γ ^* of the parameters α _s , β, γ of the nonlinear support vector machine is generated and written into the hard disk device 17. FIG. 4B is an explanatory diagram showing the configuration of the data file output in S250.

  When this process is finished, the CPU 11 proceeds to S260, and the intermediate layer output value Z = U = {u1,..., Uj, from the input value X = {x1,..., Xi,. ..., uN} conversion formula

及び、設計した非線形サポートベクタマシン（識別関数）

And designed non-linear support vector machine (discriminant function)

を記したモデル導出結果出力画面を、表示装置２１を通じて表示し、その後、当該モデル導出処理を終了する。

Is displayed through the display device 21, and then the model derivation process is terminated.

  Note that the calculation model derivation result corresponding to the learning data D (1),..., D (S) output from the model derivation device 1 in this way is used by the designer of the recognition system. That is, according to this derivation result, the recognition system calculates the output value Z of the intermediate layer one layer before the output layer in the neural network from the input value X, and uses this output value Z as the input value U of the nonlinear support vector machine. The output value y of the nonlinear support vector machine is calculated, and the category corresponding to the input value X is recognized.

  FIG. 8 is a block diagram showing a configuration example of the recognition device 30 on which the calculation model derived by the model derivation device 1 is mounted. The recognition device 30 includes a feature extraction unit 31, a recognition unit 33, and an output interface 35. When a recognition target pattern such as audio data or image data is input from the outside, the recognition unit 30 recognizes the feature extraction unit 31. The feature of the target pattern is extracted to generate an N1-dimensional feature vector, which is input to the recognition unit 33 as X = {x1,..., XN1}.

  Further, when the feature vector X = {x1,..., XN1} is input from the feature extraction unit 31, the recognition unit 33 performs the following recognition process. FIG. 9 is a flowchart showing a recognition process executed by the recognition unit 33.

  When the recognition process is started, the recognition unit 33 first converts the input value X = {x1,..., XN1} into an N-dimensional vector U = {u1,..., UN} (S310). However, the value uj of the j-th element of the vector U takes the following value.

尚、Ｗａ^*（ｉ，ｊ）は、定数であり、設計段階でモデル導出装置１により算出された学習パラメータＷａ（ｉ，ｊ）の解に対応するものである。また、ここでは、ｘ０＝１として、値ｕｊを算出する。

Wa ^* (i, j) is a constant and corresponds to the solution of the learning parameter Wa (i, j) calculated by the model deriving device 1 at the design stage. Here, the value uj is calculated assuming x0 = 1.

  When this process is finished, the recognition unit 33 calculates a value y representing the category by the following equation based on the calculated vector U (S320).

尚、値Ｓ，β^*，γ^*及びα_S ^*，ｙ（ｓ），Ｕ（ｓ）（ｓ＝１，２，…，Ｓ）も同様に、定数であり、上述の手法により算出された解α_S ^*，β^*，γ^*及び解の導出に用いられた学習データＤｖ（１），…，Ｄｖ（Ｓ）に対応した値に、設計段階で設定されるものである。

The values S, β ^* , γ ^* and α _S ^* , y (s), U (s) (s = 1, 2,..., S) are also constants, and are calculated by the above-described method. The values corresponding to the solutions α _S ^* , β ^* , γ ^* and the learning data Dv (1),..., Dv (S) used for derivation of the solutions are set at the design stage.

  When the process of S320 is completed, the recognition unit 33 outputs the value y representing the calculated category as an input pattern recognition result through the output interface 35 (S330). Thereafter, the recognition process is terminated. In this way, the value y representing the category output from the recognition device 30 is input to the subsequent information processing device (not shown), and the processing corresponding to the recognition result is performed by the information processing device. Executed.

  The configuration of the model deriving device 1 and the recognition device 30 according to the first embodiment has been described above. In this embodiment, a calculation model combining a neural network and a nonlinear support vector machine is derived as a calculation model of the recognition system. I did it. Specifically, in order to derive a suitable calculation model for an arbitrary recognition target, first, the sample X (s) of the input value X indicated by each of the S pieces of learning data D (s) and the teacher signal y (s ), The neural network is learned, and the learned neural network converts the sample X (s) into a value Z that can be easily linearly separated to generate new learning data Dv (s). Then, by designing a nonlinear support vector machine based on the learning data Dv (s), a nonlinear support vector machine capable of separating samples into categories can be obtained, and a category corresponding to the input value X can be accurately recognized. A calculation model was derived.

  In the conventionally known neural network learning method, only a local solution can be obtained as a solution of the learning parameter W1, and thus there is no theoretical guarantee that an optimal neural network can be configured. Since the calculation model is derived using a non-linear support vector machine capable of obtaining an optimal solution, an optimal recognition system calculation model can be derived based on the learning data.

  In addition, in the neural network, when there is an input outside the learning data, this cannot be recognized sufficiently accurately, but in this example, since a calculation model is derived using a nonlinear support vector machine, It is possible to derive a calculation model capable of obtaining a good recognition result even for pattern input outside the learning data.

  Further, the conventional nonlinear support vector machine design method has a problem that a suitable nonlinear support vector machine cannot be designed unless the sample X (s) originally exhibits high linear separation. In the model derivation device 1 of the example, the nonlinear support vector machine is designed by replacing the input value X (s) with a value Z (s) that can be easily linearly separated. Vector machine techniques can be employed.

  In addition, in this embodiment, since the kernel parameter γ used in the nonlinear support vector machine is also learned based on the learning data Dv (s), the value of the parameter γ is determined by the designer through trial and error as in the past. Therefore, the recognition system can be designed efficiently.

  Next, the model derivation device 1 and the recognition device 30 of the second embodiment will be described. The model derivation device 1 of the second embodiment is different from the model derivation device 1 of the first embodiment in the contents of the model derivation process and new learning data generation process executed by the CPU 11, and the recognition device of the second embodiment. 30 is the extent to which the content of the recognition process executed by the recognition unit 33 is different from the recognition device 30 of the first embodiment. Therefore, hereinafter, the contents of the different processes will be described with reference to FIGS. 10 and 11.

  FIG. 10A is a flowchart showing a model derivation process executed by the CPU 11 in the second embodiment, and is a diagram showing an extracted part of the latter half of the process. FIG. 10B is a flowchart showing new learning data generation processing executed by the CPU 11 in the second embodiment.

  In the second embodiment, when starting the model derivation process, the CPU 11 executes the processes from S110 to S210 as in the first embodiment, and in S400 thereafter, the new learning data shown in FIG. Execute the generation process.

When the new learning data generation process is started, the CPU 11 first sets the parameter s to a value 1 (S410). Thereafter, the output value Z (s) = {z1 (s) of the intermediate layer when the sample X (s) is input to the three-layer feedforward neural network in which the solution W1 ^* of the learning parameter W1 calculated in S210 is set. ,..., ZN2 (s)} is obtained based on the learning data D (s) (S420).

  Specifically, the output value Z (s) of the intermediate layer is calculated according to the following equation (provided that x0 (s) = 1).

また、この処理後には、パラメータｓを１加算した値に更新し（Ｓ４２５）、更新後のパラメータｓの値が学習データの総数Ｓよりも大きいか否かを判断する（Ｓ４３０）。

After this processing, the parameter s is updated to a value obtained by adding 1 (S425), and it is determined whether or not the updated parameter s is greater than the total number S of learning data (S430).

  If s ≦ S (No in S430), the process proceeds to S420. In this way, by repeatedly executing the processing of S420 to S430, in the new learning data generation processing, the sample X (s) of each learning data D (s) in the range from s = 1 to s = S. The output value Z (s) = {z1 (s),..., ZN2 (s)} of the intermediate layer when.

  When the output value Z (s) = {z1 (s),..., ZN2 (s)} of the corresponding intermediate layer is obtained for the sample X (s) of all the learning data D (s), Yes in S430. And the process proceeds to S440.

  After shifting to S440, the CPU 11 calculates the average value μ = {μ1,..., ΜN2} of the calculated Z (1),.

また、この後には、算出したμを用いて、分散共分散行列Ｃを、次式に従って算出する（Ｓ４４５）。尚、上付き文字Ｔは、転置を表すものとする。

Thereafter, the variance-covariance matrix C is calculated according to the following equation using the calculated μ (S445). The superscript letter T represents transposition.

Ｓ４４５において分散共分散行列Ｃを算出すると、ＣＰＵ１１は、この行列Ｃの固有値λｒ及び固有ベクトルＪｒ（ｒ＝１，２，…）を算出し（Ｓ４５０）、固有値λｒの大きい固有ベクトルＪｒから順にＮ３個の固有ベクトルＪｒを、主軸Ｊ１，…，Ｊｍ，…，ＪＮ３に設定する（Ｓ４５５）。尚、主軸に設定する固有ベクトルの数Ｎ３は、２以上でＺ（ｓ）の次数Ｎ２よりも小さい値に設定される。具体的に、主軸に設定する固有ベクトルの数Ｎ３は、Ｚ（ｓ）の次数Ｎ２よりも所定数小さい値として設計段階で予め定められてもよいし、得られた固有値λｒ（ｒ＝１，２，…）から最適な値に設定するようにしてもよい。

When the variance-covariance matrix C is calculated in S445, the CPU 11 calculates eigenvalues λr and eigenvectors Jr (r = 1, 2,...) Of the matrix C (S450), and N3 pieces in order from the eigenvector Jr having the largest eigenvalue λr The eigenvector Jr is set to the principal axes J1,..., Jm,..., JN3 (S455). Note that the number N3 of eigenvectors set for the main axis is set to a value of 2 or more and smaller than the order N2 of Z (s). Specifically, the number N3 of eigenvectors set on the main axis may be predetermined in the design stage as a value smaller than the order N2 of Z (s) at the design stage, or the obtained eigenvalues λr (r = 1, 2). ,...) May be set to an optimum value.

  When the processing in S455 is completed in this way, the CPU 11 proceeds to S460, sets the parameter N to the number N3 of spindles, and N = N3-dimensional parameter U (s) (s = 1,..., S ) Is generated (S460). Further, the parameter s is set to a value 1 (S465).

  Thereafter, Z (s) calculated in the process of S420 is converted into an N3-dimensional vector V = {v1,..., Vm,..., VN3} using the main axes J1,. ). Note that the value vm of the m-th element in this vector V is calculated by the following equation.

また、このようにして、ベクトルＺ（ｓ）を、ベクトルＶに変換すると、ＣＰＵ１１は、算出した値Ｖを、パラメータＵ（ｓ）＝｛ｕ１（ｓ），…，ｕＮ（ｓ）｝に設定し、新たな学習データＤｖ（ｓ）＝｛Ｕ（ｓ）＝Ｖ，ｙ（ｓ）｝を生成する（Ｓ４７５）。その後、パラメータｓを１加算した値に更新し（Ｓ４８０）、更新後のパラメータｓの値が学習データの総数Ｓよりも大きいか否かを判断する（Ｓ４９０）。

Further, when the vector Z (s) is converted into the vector V in this way, the CPU 11 sets the calculated value V to the parameter U (s) = {u1 (s),..., UN (s)}. Then, new learning data Dv (s) = {U (s) = V, y (s)} is generated (S475). Thereafter, the parameter s is updated to a value obtained by adding 1 (S480), and it is determined whether or not the updated parameter s is greater than the total number S of learning data (S490).

  If s ≦ S (No in S490), the process proceeds to S470. In this way, by repeatedly executing the processing of S470 to S490, in the new learning data generation processing, Z calculated based on each learning data D (s) in the range from s = 1 to s = S. Learning data Dv (s) is generated from (s). When generation of new learning data Dv (s) has been completed for all learning data D (s), it is determined Yes in S490, and the new learning data generation processing ends.

When the new learning data generation process in S400 is completed, the CPU 11 proceeds to S500 and executes the kernel parameter setting process as in the first embodiment to obtain a solution γ ^{* of the} parameter γ. Thereafter, the process proceeds to S510.

In S510, the CPU 11 designs a nonlinear support vector machine using the learning data Dv (1),..., Dv (S) and the solution γ ^{* of the} parameter γ. That is, similarly to S240, the constrained secondary optimization problem is solved, and the solutions α _s ^* and β ^* of the unknown parameters α _s and β in the nonlinear support vector machine are obtained.

In this manner, when the solutions α _s ^* and β ^* of the parameters α _s and β of the nonlinear support vector machine are calculated in S510, the CPU 11 proceeds to S520, and the learning parameter Wa (i, j calculated in S210 described above is obtained. ) solution Wa ^* (i, j), i.e., solutions Wa ^* (i learning parameters required for calculating the output value Z of the intermediate layer Wa (i, j), j), and, from the vector Z to the vector V A data file describing the solutions α _s ^* , β ^* , γ ^* of the parameters α _s , β, γ of the nonlinear support vector machine designed in the main axes J1,. To the hard disk device 17. Specifically, in S520, in addition to the description shown in FIG. 4B, the values of N3 spindles J1, J2,..., JN3 are described to generate a data file, which is stored in the hard disk device. 17 is written.

  When this process is finished, the CPU 11 proceeds to S530, and the vector V = U = {u1,..., Um,..., UN} described above from the input value X = {x1,. Conversion formula to

及び、Ｓ５１０で設計した非線形サポートベクタマシンを記したモデル導出結果出力画面を、表示装置２１を通じて表示し、その後、当該モデル導出処理を終了する。

And the model derivation result output screen which described the nonlinear support vector machine designed by S510 is displayed through the display apparatus 21, and the said model derivation process is complete | finished after that.

  The model derivation process of the second embodiment has been described above. In the second embodiment, based on this output result, the output value Z of the intermediate layer one layer before the output layer of the neural network is calculated from the input value X. Then, the dimension of the output value Z is reduced using N3 N2-dimensional vectors Jm, and the output value y of the nonlinear support vector machine is calculated using the dimension-reduced value V as the input value U of the nonlinear support vector machine. The calculation model is mounted on the recognition device 30.

  Subsequently, a recognition process executed by the recognition unit 33 of the recognition device 30 according to the second embodiment on which the calculation model is mounted will be described. FIG. 11 is a flowchart showing a recognition process executed by the recognition unit 33 of the second embodiment.

  When the feature vector X is input from the feature extraction unit 31 and the recognition process illustrated in FIG. 11 is started, the recognition unit 33 first converts the input value X = {x1,..., XN1} into an N2-dimensional vector Z = {z1. ,..., ZN2} (S610). However, the value zj of the j-th (j = 1, 2,..., N2) element of the vector Z takes the following values.

尚、Ｗａ^*（ｉ，ｊ）は、定数であり、設計段階でモデル導出装置１により算出された学習パラメータＷａ（ｉ，ｊ）の解に対応するものである。また、ここでは、ｘ０＝１として、値ｚｊを算出する（Ｓ６１０）。

Wa ^* (i, j) is a constant and corresponds to the solution of the learning parameter Wa (i, j) calculated by the model deriving device 1 at the design stage. Here, the value zj is calculated with x0 = 1 (S610).

  When this processing is completed, the recognition unit 33 reduces the dimension of the calculated vector Z according to the following equation and converts it into an N-dimensional vector U = {u1,..., Um,..., UN} (S620). . The vector Jm corresponds to the main axis calculated by the model deriving device 1 at the design stage.

その後、算出したベクトルＵに基づき、次式により、カテゴリを表す値ｙを算出する（Ｓ６３０）。

Thereafter, based on the calculated vector U, a value y representing the category is calculated by the following equation (S630).

尚、値Ｓ，β^*，γ^*及びα_S ^*，ｙ（ｓ），Ｕ（ｓ）（ｓ＝１，２，…，Ｓ）も同様に、定数であり、上述の手法により算出された解α_S ^*，β^*，γ^*及び解の導出に用いられた学習データＤｖ（１），…，Ｄｖ（Ｓ）に対応した値に、設計段階で設定されるものである。

The values S, β ^* , γ ^* and α _S ^* , y (s), U (s) (s = 1, 2,..., S) are also constants, and are calculated by the above-described method. The values corresponding to the solutions α _S ^* , β ^* , γ ^* and the learning data Dv (1),..., Dv (S) used for derivation of the solutions are set at the design stage.

  When the processing of S630 is completed, the recognition unit 33 outputs the value y representing the calculated category through the output interface 35 as the recognition result of the input pattern (S640). Thereafter, the recognition process is terminated.

  The operation of the model derivation device 1 and the recognition device 30 according to the second embodiment has been described above. According to this embodiment, the pattern recognition is performed with a small amount of computation in order to reduce the dimension of the output value Z of the intermediate layer of the neural network. A calculation model of a possible recognition system can be derived, which is very convenient.

  Note that the acquisition means included in the information processing apparatus of the present invention is realized by the processing of S110 to S200 in the above embodiment, and the learning means is realized by the processing of S210. Further, the new learning data generating means is realized by the process of S220 in the first embodiment, and is realized by the process of S400 in the second embodiment.

  Further, the support vector machine design means is realized by the processes of S230 and S240 in the first embodiment, and is realized by the processes of S500 and S510 in the second embodiment. In addition, an output means is implement | achieved by the process of S250-S260 in a 1st Example, and is implement | achieved by the process of S520-S530 in a 2nd Example.

  In addition, the acquisition means provided in the design apparatus of the present invention is realized by the process of S220 in the first embodiment, and is realized by the process of S400 in the second embodiment. The appropriate value calculation means is realized by the kernel parameter setting process, and the design means is realized by the process of S240 in the first embodiment, and is realized by the process of S510 in the second embodiment.

Further, the present invention is not limited to the above-described embodiments, and can take various forms.
For example, in the above-described embodiment, the nonlinear support vector machine using the Gaussian kernel is designed, but the nonlinear support vector machine may be designed using another kernel such as a polynomial kernel. Further, a non-linear support vector machine may be designed without using a kernel.

  When designing a non-linear support vector machine without using a kernel, for example,

として、１以上の整数の組合せ｛ｅ１（ｎ），ｅ２（ｎ），…，ｅＮ（ｎ）｝を、ｎ＝１，２，…，Ｎ５において組合せが重複しないように、Ｎ５個採り、Ｎ次元のベクトルＵ＝｛ｕ１，…，ｕＮ｝を、それより高次元のＮ５次元のベクトルφ（Ｕ）＝｛ｂ１，…，ｂＮ５｝に置換する写像φを定めればよい。勿論、このようにモデル導出装置１を構成する場合には、カーネルパラメータ設定処理を実行しないように、モデル導出処理を構成すればよい。

Assuming that N5 integer combinations {e1 (n), e2 (n),..., EN (n)} are taken so that the combinations do not overlap at n = 1, 2,. A mapping φ that replaces the dimensional vector U = {u1,..., UN} with a higher-dimensional N5 dimensional vector φ (U) = {b1,. Of course, when the model deriving device 1 is configured in this way, the model deriving process may be configured not to execute the kernel parameter setting process.

  In addition, in the processes of S330 and S640, the recognition device 30 may be configured to output information indicating the accuracy of the recognition result (value y) together with the value y indicating the category through the output interface 35.

  Specifically, as the information representing the accuracy of the recognition result, the absolute value | p | of the input value p to the sign function constituting the nonlinear support vector machine obtained in S320 and S630 when the value y is calculated is adopted. can do.

上述したように、符号関数への入力値ｐは、超平面からの符号付距離に対応する値である。従って、入力値ｐの絶対値が大きければ、認識結果の確度が高いということができ、入力値ｐの絶対値が小さければ、認識結果の確度が低いということができる。

As described above, the input value p to the sign function is a value corresponding to the signed distance from the hyperplane. Therefore, it can be said that the accuracy of the recognition result is high if the absolute value of the input value p is large, and the accuracy of the recognition result is low if the absolute value of the input value p is small.

  Therefore, if the recognition apparatus 30 is configured to output such information (value | p |) indicating the accuracy together with the recognition result (value y), the information processing apparatus that is the output destination uses the information indicating the accuracy. If it is possible to determine whether the recognition process needs to be performed again and the accuracy is low, the recognition process can be performed again by, for example, causing the user to utter the voice as the recognition target.

2 is a block diagram illustrating a configuration of a model derivation device 1. FIG. It is a flowchart showing the model derivation process which CPU11 performs. It is a basic composition figure (a) of a calculation model derived by model derivation processing, and a composition figure (b) (c) of the conventional calculation model. It is explanatory drawing showing the structure of the data file input / output. It is explanatory drawing which showed the correspondence of a parameter and a coupling coefficient. It is a flowchart showing the new learning data generation process which CPU11 performs. It is a flowchart showing the kernel parameter setting process which CPU11 performs. 3 is a block diagram illustrating a configuration example of a recognition device 30. FIG. It is a flowchart showing the recognition process which the recognition part 33 performs. It is the flowchart (a) showing the model derivation process which CPU11 performs in a 2nd Example, and the flowchart (b) showing new learning data generation processing. It is a flowchart showing the recognition process which the recognition part 33 performs in 2nd Example. It is explanatory drawing (a) which showed the example of the calculation method by a neural network, and explanatory drawing (b) which showed the problem of the neural network.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Model deriving device, 11 ... CPU, 13 ... ROM, 15 ... RAM, 17 ... Hard disk device, 21 ... Display device, 23 ... User interface, 25 ... Drive device, 30 ... Recognition device, 31 ... Feature extraction part, 33 ... recognition unit, 35 ... output interface

Claims (17)

From a given input value X = {x1,..., XN ₁ } (where the value N ₁ is an integer of 2 or more), a value y representing a category corresponding to the input value X is calculated in a predetermined manner. calculated by the model, the calculation model of recognizing systems the category corresponding to the input value X, the input value X of the sample X (s) and the value y representing the category to which the sample belongs (s) A method of deriving based on arbitrary S pieces of learning data D (s) = {X (s), y (s)} (where s = 1,..., S) composed of combinations. ,
As a learning target neural network, a (L ₁ +1) layer neural network in which the 0th layer is an input layer composed of N ₁ input units and the L ₁ layer is an output layer (however, the value L ₁ is 2 or more) In this neural network, an unknown number of learning parameters W ₁ are used as the training signal D by using the value y (s) of each learning data D (s) as a teacher signal. A procedure [a] for learning based on (s) and obtaining a solution of the learning parameter W ₁ ;
New learning corresponding to each learning data D (s) is performed using the neural network of the (L ₁ +1) layer after learning obtained by setting the solution of the learning parameter W ₁ obtained in the procedure [a]. A procedure for generating data D _v (s), for each learning data D (s), a sample X (s) of learning data D (s) is stored in the (L ₁ +1) layer after learning. As an input value X to the neural network, an output value Z of the (L ₁ −1) -th layer composed of N ₂ (the value N ₂ is an integer of 2 or more) intermediate units constituting this neural network. = {Z1,..., ZN ₂ }, the obtained output value Z is set as a new sample U (s), the set new sample U (s), and the learning data D (s) a combination of the values y (s) indicated by the data D _v (s) {U (s), y ( s)} a, the procedure [b] generating said as learning data D new learning data corresponding to the _{(s) D v (s)} ,
Based on the learning data D _v (s) generated in the procedure [b], the input value U = {u1,..., UN} (where the value N is the learning data D _{v used} as a sample of the input value U). (C is the number of dimensions of the sample U (s) in (s).) A procedure [c] for designing a nonlinear support vector machine that calculates a value y representing a category corresponding to
Including
A combination of a calculation model capable of calculating the output value Z of the (L ₁ -1) layer in the neural network of the (L ₁ +1) layer after learning and the nonlinear support vector machine designed in the step [c] The output model Z of the (L ₁ −1) layer is calculated from the input value X, and the output value Z is used as the input value U of the nonlinear support vector machine. A model derivation method for deriving a calculation model for calculating an output value y as a calculation model of the recognition system
The value y representing the category corresponding to the input value X is calculated from the given input value X = {x1,..., XN ₁ } using the calculation model derived by A recognition system characterized by recognizing a category to be performed. The model derivation method is:
In the step [b], for each learning data D (s), the value Z calculated based on the sample X (s) of the learning data D (s) is reduced in dimension by a predetermined algorithm, and the dimension The value V = {v1,..., VN ₃ } after reduction (where the value N ₃ is an integer of 2 or more smaller than the value N ₂ ) is set as the new sample U (s), and Learning data D _v (s) = {U (s), y (s)} is generated,
As a calculation model of the recognition system,
The output value Z of the (L ₁ −1) -th layer is calculated from the input value X, the dimension of the output value Z is reduced by the predetermined algorithm, and the output value Z is converted into an N ₃ -dimensional value V = { v1,..., vN ₃ }, and a model derivation method for deriving a calculation model for calculating the output value y of the nonlinear support vector machine using the converted value V as the input value U of the nonlinear support vector machine. recognition system according to claim 1, wherein a. The model derivation method is:
In the step [b], for each learning data D (s), the value Z calculated based on the sample X (s) of the learning data D (s) is obtained by the principal component analysis method N _3. number of N _{2-dimensional} vector Jm (where, m = 1, ..., a N _3.) by dimension reduction using, the value Z, the value of the m element vm is said value Z vector Jm inner product <Jm, Z> value of N _{3-dimensional} represented by V = {v1, ..., vN _3} is converted to a value V = after the dimension reduction {v1, ..., vN _3}, said new The learning data D _v (s) = {U (s), y (s)} is set as a simple sample U (s),
As a calculation model of the recognition system,
The output value Z of the (L ₁ −1) -th layer is calculated from the input value X, the output value Z is dimension-reduced using the N ₃ N _2- dimensional vectors Jm, and the value V after dimension reduction is obtained. The recognition system according to claim 2, wherein the model is a model derivation method for deriving a calculation model for calculating an output value y of the nonlinear support vector machine with the input value U of the nonlinear support vector machine. The model derivation method is:
4. The recognition system according to claim 1, wherein the procedure [a] is a model derivation method for obtaining a solution of the learning parameter W ₁ by a back-propagation method. 5. The model derivation method is:
In the procedure [c], for the nonlinear support vector machine that calculates the value y representing the category from the input value U, the parameter γ of the kernel K (U, U (s), γ) constituting the nonlinear support vector machine is changed. The nonlinear support vector machine is designed using the kernel K (U, U (s), γ) in which the appropriate value is obtained by the following procedure [1] to procedure [3] and the appropriate value is set as the parameter γ. The recognition system according to claim 1, wherein the recognition system is a model derivation method .
[1] As a learning target neural network, a (L ₂ +1) layer neural network in which the 0th layer is an input layer made up of N input units and the L ₂ layer is an output layer (where the value L ₂ is And an unknown number of learning parameters W ₂ in the neural network, the value y (s) of each learning data D _v (s) as a teacher signal, and the S number of learning parameters W ₂ . Learning is performed based on the learning data D _v (s), and a solution of the learning parameter W ₂ is obtained.
[2] Learning obtained by setting a solution of the learning parameter W ₂ obtained in the procedure [1] for a sample U (s) of the learning data D _v (s) for each learning data D _v (s). As input values to the neural network of the later (L ₂ +1) layer, N _h pieces constituting the neural network (however, the value N _h is an integer larger than the dimension number N of the input value U). The output value H (s) = {h1 (s),..., HN _h (s)} of the (L ₂ −1) -th layer consisting of the intermediate units is obtained.
[3] Using the learning data D _v (s) and the value H (s) corresponding to the sample U (s) of each learning data D _v (s) obtained in the procedure [2],

Accordingly, a solution γ ^{* of the} parameter γ for which the square error E takes a minimum value is obtained as an appropriate value of the parameter γ (where <H (s = a), H (s = b)> is a value H (s = A) represents the inner product of the value H (s = b)). The kernel K (U, U (s), γ) is a Gaussian kernel.

The recognition system according to claim 5, wherein: As a recognition result of the category corresponding to the input value X, a value y representing the category corresponding to the input value X is output, and the code constituting the nonlinear support vector machine obtained when the value y is calculated 7. The recognition system according to claim 1, wherein the absolute value | p | of the input value p to the function is output as information indicating the accuracy of the recognition result. . From a given input value X = {x1,..., XN ₁ } (where the value N ₁ is an integer of 2 or more), a value y representing a category corresponding to the input value X is calculated in a predetermined manner. An information processing apparatus used for deriving the calculation model of a recognition system that recognizes a category corresponding to the input value X calculated by a model,
S learning data D (s) = {X (s), y (s)} consisting of a combination of the sample X (s) of the input value X and the value y (s) representing the category to which the sample belongs (provided that , S = 1,..., S.)
A (L ₁ +1) layer neural network in which the 0th layer is configured as an input layer composed of N ₁ input units and the L ₁ layer is configured as an output layer (however, the value L ₁ is an integer of 2 or more) S) learning data D acquired by the acquisition means using the unknown learning parameter W ₁ as the teacher signal and the value y (s) of each learning data D (s) acquired by the acquisition means. Learning means for learning based on (s) and obtaining a solution of the learning parameter W ₁ ;
Corresponding to each learning data D (s) acquired by the acquisition unit using the neural network of the (L ₁ +1) layer after learning by setting the solution of the learning parameter W ₁ obtained by the learning unit New learning data D _v (s) to be generated, and for each learning data D (s), a sample X (s) of the learning data D (s) is converted to the (L ₁ after learning). As an input value X to the neural network of the +1) layer, the (L ₁ -1) th layer composed of N ₂ (the value N ₂ is an integer of 2 or more) intermediate units constituting this neural network. Output value Z = {z1,..., ZN ₂ }, set the obtained output value Z as a new sample U (s), the set new sample U (s), and the learning data From the combination of the values y (s) indicated by D (s) Data D _v (s) = that the new data generating means for generating {U (s), y ( s)} a, as the learning data D (s) corresponding to the new training data D _v (s),
Based on the learning data D _v (s) generated by the new data generating means, the input value U = {u1,..., UN} (where the value N is the learning data D that is a sample of the input value U). _v is the number of dimensions of the sample U (s) of (s).) Support vector machine design means for designing a nonlinear support vector machine that calculates a value y representing a category corresponding to
Output means for outputting information representing the solution of the learning parameter W ₁ obtained by the learning means, and information representing the nonlinear support vector machine designed by the support vector machine design means;
An information processing apparatus comprising: For each learning data D (s), the new data generating means reduces the value Z calculated based on the sample X (s) of the learning data D (s) by a predetermined algorithm, The value V = {v1,..., VN ₃ } after reduction (where the value N ₃ is an integer of 2 or more smaller than the value N ₂ ) is set as the new sample U (s), and Learning data D _v (s) = {U (s), y (s)} is generated,
The output means includes information representing the solution of the learning parameter W ₁ obtained by the learning means, information representing the nonlinear support vector machine designed by the support vector machine design means, and the value Z 9. The information processing apparatus according to claim 8 , wherein the information processing apparatus is configured to output information indicating a conversion method to the value V. It said new data generating means, wherein each training data D (s), the value Z calculated on the basis of the samples X (s) of the training data D (s), N ₃ obtained by the technique of principal component analysis number of N _{2-dimensional} vector Jm (where, m = 1, ..., a N _3.) by dimension reduction using, the value Z, the value of the m element vm is said value Z vector Jm inner product <Jm, Z> value of N _{3-dimensional} represented by V = {v1, ..., vN _3} is converted to a value V = after the dimension reduction {v1, ..., vN _3}, said new Set as a simple sample U (s) to generate the learning data D _v (s) = {U (s), y (s)},
The output means is configured to output N ₃ N _two- dimensional vectors Jm obtained by a principal component analysis method as information representing a conversion method from the value Z to the value V. The information processing apparatus according to claim 9 . The information processing apparatus according to claim 8 , wherein the learning unit is configured to obtain a solution of the learning parameter W ₁ by a back propagation method. The support vector machine design means calculates the parameter y of the kernel K (U, U (s), γ) constituting the nonlinear support vector machine for the nonlinear support vector machine that calculates the value y representing the category from the input value U. Is determined by the following procedure [1] to procedure [3], and the non-linear support vector machine is obtained by using the kernel K (U, U (s), γ) in which the appropriate value is set as the parameter γ. The information processing apparatus according to claim 8 , wherein the information processing apparatus is configured to be designed.
[1] As a learning target neural network, a (L ₂ +1) layer neural network in which the 0th layer is an input layer made up of N input units and the L ₂ layer is an output layer (where the value L ₂ is And an unknown number of learning parameters W ₂ in the neural network, the value y (s) of each learning data D _v (s) as a teacher signal, and the S number of learning parameters W ₂ . Learning is performed based on the learning data D _v (s), and a solution of the learning parameter W ₂ is obtained.
[2] Learning obtained by setting a solution of the learning parameter W ₂ obtained in the procedure [1] for a sample U (s) of the learning data D _v (s) for each learning data D _v (s). As input values to the neural network of the later (L ₂ +1) layer, N _h pieces constituting the neural network (however, the value N _h is an integer larger than the dimension number N of the input value U). The output value H (s) = {h1 (s),..., HN _h (s)} of the (L ₂ −1) -th layer consisting of the intermediate units is obtained.
[3] Using the learning data D _v (s) and the value H (s) corresponding to the sample U (s) of each learning data D _v (s) obtained in the procedure [2],

Accordingly, a solution γ ^{* of the} parameter γ for which the square error E takes a minimum value is obtained as an appropriate value of the parameter γ (where <H (s = a), H (s = b)> is a value H (s = A) represents the inner product of the value H (s = b)). The kernel K (U, U (s), γ) is a Gaussian kernel.

The information processing apparatus according to claim 12, wherein: 14. The function as the acquisition means, the learning means, the new data generation means, the support vector machine design means, and the output means included in the information processing apparatus according to claim 8 is a computer. A program to make it happen. This is a design apparatus for a nonlinear support vector machine that calculates a value y representing a category of the input value U from the input value U = {u1,..., UN} (where the value N is an integer of 2 or more). And
S learning data D _v (s) = {U (s), y (s)} () consisting of a combination of a sample U (s) of the input value U and a value y (s) representing the category to which the sample belongs. Where s = 1,..., S.)
Based on the S learning data D _v (s) acquired by the acquisition means, an appropriate value of the parameter γ of the kernel K (U, U (s), γ) employed in the nonlinear support vector machine is determined by the following procedure. [1] to an appropriate value calculating means obtained by the procedure [3];
Based on the S pieces of learning data D _v (s) acquired by the acquisition unit using the kernel K (U, U (s), γ) in which appropriate values of the obtained parameters γ are set, the nonlinear support vector Design means to design the machine;
Output means for outputting information representing the nonlinear support vector machine designed by the design means;
A design apparatus comprising:
[1] As a learning target neural network, a (L + 1) layer neural network in which the 0th layer is an input layer composed of N input units and the Lth layer is an output layer (where the value L is an integer of 2 or more) in a.) set the learning parameters W unknowns in this neural network, the value y (s) of the training data D _v (s) as a teacher signal, the S pieces of learning data D _v (s) And learning of the learning parameter W is obtained.
[2] After learning, a sample U (s) of learning data D _v (s) is set for each learning data D _v (s) as a solution of the learning parameter W obtained in step [1]. As an input value to the (L + 1) layer neural network, N _h pieces (where the value N _h is an integer larger than the dimension number N of the input value U) constituting the neural network. The output value H (s) = {h1 (s),..., HN _h (s)} of the (L−1) th layer is obtained.
[3] Using the learning data D _v (s) and the value H (s) corresponding to the sample U (s) of each learning data D _v (s) obtained in the procedure [2],

Accordingly, a solution γ ^{* of the} parameter γ for which the square error E takes a minimum value is obtained as an appropriate value of the parameter γ (where <H (s = a), H (s = b)> is a value H (s = A) represents the inner product of the value H (s = b)). The kernel K (U, U (s), γ) is a Gaussian kernel.

The design apparatus according to claim 15, wherein: A program for causing a computer to realize the functions of the acquisition unit, the appropriate value calculation unit, the design unit, and the output unit included in the design apparatus according to claim 15 or 16 .

Images