JP6101650B2 - System parameter learning apparatus, information processing apparatus, method, and program - Google Patents

Description

  The present invention relates to a system parameter learning apparatus, information processing apparatus, method, and program, and more particularly, to a system parameter learning apparatus, information processing apparatus, method, and program for learning system parameters.

  As shown in FIG. 13, identification problems in information processing such as speech recognition, machine translation, character recognition, object recognition, DNA structure prediction, etc. predict output when input is given as shown in FIG. System.

  These systems can generally be divided into an execution phase and a construction phase. The construction phase refers to the work of designing a system in advance by hand and determining system parameters and the like. The execution phase processes the input based on the design defined in the construction phase, and the output is determined depending on the system parameters.

  In the construction phase, the system can be constructed in various ways. For example, a method is conceivable in which a conversion rule is described manually, an input is converted into an output in accordance with the rule, and the output is output. However, manual preparation of conversion rules is very expensive to maintain completeness and consistency. Therefore, as shown in FIG. 15, a machine learning method for automatically constructing a system from data is used. In recent years, the method of automatically constructing a system has been mainstream.

  In the construction phase, first, an input pair of the target system and a corresponding output pair are prepared. This is generally called correct answer data or teacher data. Teacher data is data representing what kind of output should be made when input in teacher data is input to the system. Next, a system is constructed using the teacher data. The necessary requirement is that the system can perform correct output with respect to the input in the teacher data. Therefore, in the construction phase based on machine learning, the teacher data is used to result in learning a set of system parameters that can correctly discriminate the teacher data.

  The above processing is expressed mathematically as follows. First, the execution phase is shown. Let x ^ represent one input and let Χ be the set of all inputs x ^ that the system can accept. Note that “^” attached to a symbol indicates that the symbol is a matrix, a multidimensional array, or a vector. Similarly, let y denote a single output, and let Y be a set of all possible outputs y that the system allows. Also, let Y (x ^) be a set of all outputs y ^ that can be taken when x ^ is given. Therefore, the relationship of x ^ ∈Χ and y ^ ∈Y (x ^) ⊆Y holds.

Next, let w ^ be a vector representation of a set of system parameters. Here, w _d is the d-th element of the vector w ^, and at the same time is the d-th system parameter. That is, w ^ = (w ₁ ,..., W _N ) and d = {1,. . . , N} holds. However, the number of system parameters is N, and w ^ is an N-dimensional vector.

  At this time, a system that returns an output y ^ when an input x ^ is given can be expressed by the following equation (1).

However, Φ (x ^, y ^: w ^) is a function that determines a score when converting from x ^ to y ^, and is simply referred to as a score function here. That is, among all the outputs y ^ that may be obtained when x ^ is given, y ^ having the highest conversion score is adopted as the output. Therefore, w ^ is a system parameter that controls which output is selected, and can be said to be a factor that determines the performance of the entire system. Therefore, how to obtain the system parameter w ^ with high accuracy is the greatest requirement in the construction phase. Here, the term “accurately” means to obtain w ^ that can perform as many correct outputs as possible for every input. Note that “ ^* ” added in front of a symbol indicates that the symbol is an estimated value.

  Next, the construction phase will be described. In fact, it is very difficult to find the best parameter w for every possible input. That is due to the fact that it is virtually difficult to enumerate all possible inputs. Therefore, in the field of pattern recognition, w ^ is determined based on actual data. First, the teacher data

Represented by The teacher data is composed of a set of pairs of input x ^ and output y ^. That means

At this time, x ^ _i is the i-th input data in the teacher data, and y ^ _i is the output corresponding to the i-th input. Learning system parameters can be obtained by solving the optimization problem of the following equation (2).

  At this time,

Is a function called a risk function or a loss function, and returns a value indicating how much correct output can be obtained with respect to the input in the teacher data. When the current parameter w is used to actually make a determination using the above equation (1), if more mistakes are made, a function having a larger value is used. Ω (w ^) is generally called a regularization term, and in a situation where there is only a limited number of teacher data, system parameters are over-adapted to teacher data so that it can be correctly identified even for data that does not appear in the teacher data. This is a term that gives a penalty. For example, it is often used to restrict a parameter from taking an extremely large value by imposing a constraint such that the L ₂ -norm of the parameter is as small as possible. Finally, it can be said that ^* w ^ obtained by the above equation (2) is a set of parameters that can best identify the teacher data.

  The above is a mathematical definition of the execution phase and the construction phase of the information processing system targeted by the present invention.

Acquisition of system parameters based on the above equation (2) is called supervised learning in pattern recognition. At this time, the learned system parameter ^* w ^ is represented by a real value. Therefore, the value of ＾ is written to a file or the like at the end of the construction phase, and in the execution phase, the written file is read to calculate a score function and obtain an output. That is, the larger the number of parameters, the larger the file size required to hold the information.

  The file size is the same as the memory occupancy during execution. Memory occupancy can be a significant problem in computing environments with limited resources, such as portable terminals. In addition, even when executing on a general computer, it is desirable that the memory occupancy is as small as possible from the viewpoint of not affecting other programs as much as possible when simultaneously executing multiple on a recent multi-core computer. . In other words, in essence, it can be said that the required resource amount (file size, exclusive memory amount, etc.) for program execution is better in any calculation environment. Based on this idea, the problem of compressing the model size after learning in the learning process has been actively addressed in recent years.

The simplest method is a method of performing model compression after learning using the effect of L ₁ regularization term (non-patent document 1). The principle of this method is that the L ₁ regularization term has the effect of making the parameter zero as much as possible during learning. When the parameter becomes zero, the items related to the parameter can be deleted from the model. The size can be reduced.

  In addition, as a method that allows model compression, if multiple parameters take the same value after system parameter learning, it is possible to use this duplicate information to reduce the amount of information that must actually be retained. There is a method that uses the principle of being (Non-patent Document 2). Thus, it is a very big problem in practice to compress the learning model without reducing the accuracy of the information processing system as much as possible, and improvements have been made by various devices.

Lin Xiao. Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization. Journal of Machine Learning Research, 11: 2543-2596, 2010. Jun Suzuki and Masaaki Nagata.Supervised Model Learning with Feature Grouping based on a Discrete Constraint.Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 18-23,2013.

  However, in the technique described in Non-Patent Document 1, setting the value to zero may reduce the accuracy of the system unless the item related to the system parameter is truly meaningless. There is a trade-off between the effect of the system and the accuracy of the system. In other words, if you want to reduce the model size by increasing the number of zeros as much as possible, there is a problem that if the learning is performed with the setting of too many zeros, the system accuracy will be insufficient.

  The technique described in Non-Patent Document 2 has been shown to be very effective. On the other hand, when actually creating a model, how much can the model be compressed without reducing the accuracy of the system? It is not self-evident, but it can be tried by trial and error and finally selecting the best result from development data. Actually, as the amount of learning data increases, this work becomes very expensive and causes operational problems. As a result, it was impossible to find a high-precision and high-compression model that could have been obtained because trial and error costs could not be obtained, and as a result, a high-compression model could not be used. On the contrary, there is a problem that the accuracy is often reduced.

  The present invention has been made to solve the above problems, and an object thereof is to provide a system parameter learning apparatus, method, and program capable of automatically acquiring an appropriate high compression model.

  It is another object of the present invention to provide an information processing apparatus and program that can reduce resources required at the time of execution.

In order to achieve the above object, a system parameter learning device according to a first invention performs predetermined information processing on input data using a score function for determining a score of output data with respect to input data. A system parameter learning device for learning a plurality of system parameters set in an information processing system for outputting output data, each of a plurality of input data and a plurality of correct output data for the plurality of input data Based on a teacher data input unit that receives a pair of teacher data, the teacher data received by the teacher data input unit, and the score function, each value of the plurality of system parameters is a predetermined number of real values Included in a set of discrete values consisting of v _i (i = 1,..., ζ), real values −v _i (i = 1,..., ζ) and 0 And a learning unit that learns the optimized plurality of system parameters.

A system parameter learning method according to a second invention includes a teacher data input unit and a learning unit, and uses a score function for determining a score of output data with respect to input data. A system parameter learning method in a system parameter learning device that learns a plurality of system parameters, set in an information processing system that performs processing and outputs output data, wherein the teacher data input unit includes a plurality of input data And a plurality of correct output data for each of the plurality of input data are received, and the learning unit is based on the teacher data received by the teacher data input unit and the score function, the value of each of the plurality of system parameters is the predetermined number of real numbers _{v i (i = 1, ···} , ζ) and Numerical _{-v i (i = 1, ···} , ζ) satisfy the constraints in the set of discrete values consisting of 0 Prefecture, and learns the plurality of system parameters optimized.

According to the first and second inventions, the teacher data input unit accepts teacher data that is a pair of each of the plurality of input data and each of the plurality of correct output data for the plurality of input data, and the learning unit Based on the received teacher data and the score function, the values of the plurality of system parameters are a predetermined number of real values v _i (i = 1,..., Ζ) and real values −v _i (i = 1,..., Ζ) and 0 satisfy a constraint included in the set of discrete values and learn a plurality of optimized system parameters.

In this way, the teacher data that is a pair of each of the plurality of input data and each of the plurality of correct output data for the plurality of input data is received, and a plurality of system parameters are based on the received teacher data and the score function By learning a plurality of optimized system parameters that satisfy the constraints included in the set of discrete values consisting of a predetermined number of real values v _i , real values −v _i and 0 It is possible to automatically obtain an appropriate high compression model.

  An information processing apparatus according to a third invention is based on an input unit that receives input data, the score function, and the system parameter of the index number of each group stored by the system parameter learning apparatus of the first invention. And an information processing unit that performs the predetermined information processing on the input data received by the input unit and outputs output data.

  Moreover, the program of this invention is a program for functioning a computer as each part which comprises said system parameter learning apparatus or information processing apparatus.

As described above, according to the parameter learning apparatus, method, and program of the present invention, teacher data that is a pair of each of a plurality of input data and a plurality of correct output data for a plurality of input data is received, Based on the received teacher data and the score function, each of the plurality of system parameters has a constraint included in a set of discrete values including a predetermined number of real values v _i , real values −v _i, and 0. By learning a plurality of system parameters that are satisfied and optimized, an appropriate high compression model can be obtained automatically.

  Further, according to the information processing apparatus of the present invention, based on the system parameter of the index number of each group saved by the system parameter learning apparatus, the input data is subjected to predetermined information processing and output data is output. As a result, resources required at the time of execution can be reduced.

It is a figure which shows the example of the problem to which embodiment of this invention is applied. It is a conceptual diagram for demonstrating an example of the definition of a feature and a system. It is a block diagram which shows the structure of the system parameter learning apparatus which concerns on embodiment of this invention. It is a 1st conceptual diagram for demonstrating an example of a feature extraction function. It is a 2nd conceptual diagram for demonstrating an example of a feature extraction function. It is a conceptual diagram for demonstrating the learning process of a system parameter value based on teacher data. It is a figure which shows the example of the mapping problem to a hyperplane. It is a figure which shows the example of a process equivalent to one-dimensional k-means. It is a block diagram which shows the structure of the information processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the content of the system parameter learning process routine in the system parameter learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the content of the information processing routine in the information processing apparatus which concerns on embodiment of this invention. It is a figure which shows the experimental result in the case of using this Embodiment. It is the 1st explanatory view for explaining an outline of conventional technology. It is the 2nd explanatory view for explaining an outline of conventional technology. It is the 3rd explanatory view for explaining an outline of conventional technology.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Principle of the present invention>

  In the embodiment according to the present invention, a processing apparatus that automatically builds a high-precision and high-compression model from learning data is added to the model compression framework of Non-Patent Document 2.

First, the mechanism of the machine learning device having the model compression processing device of Non-Patent Document 2 will be described. The basic idea is that if multiple parameters have the same value after system parameter learning, this duplicate information can be used to reduce the amount of information that must actually be retained. We are using. For example, when the number of parameters is 5, the parameter set finally obtained by the above equation (2) is ^* w ^ = (0.3, 0.3, -0.6, -0.6, 1.0 ). Then, in ^* w ^, 0.3 and -0.6 appear twice, so the first and second parameters are combined, and the third and fourth parameters are combined ^. The same result can be obtained if three pieces of information such as w ^ = (0.3, -0.6, 1.0) are at least. Thus, it can be seen that as the number of duplicate values increases, it is possible to retain equivalent information with a reduced amount of information.

  In other words, when learning the system parameter of the above formula (2), the result of adding the constraint that “the value of the system parameter takes the same value as much as possible” is added and the system parameter is learned. As a result, the duplication of system parameter values obtained as follows is increased.

  Then, for the obtained parameters, the parameters having the same value are combined into one to reduce the amount of information to be held. Finally, the system is actually executed using the reduced parameters. In summary, the conventional method is roughly divided into the following three processes 1 to 3.

1. As a first process, a process for acquiring system parameters so that the parameter values finally obtained at the time of normal system parameter learning become as much as possible.
2. As a second process, a system parameter compression process for minimizing the amount of information that should be retained by holding duplicate values of the obtained system parameters together.
3. A third process is a process for operating the system in the execution phase by using the system parameter compressed to the minimum.

  More specifically, the first process replaces the system parameter learning problem of the above formula (2) with the problem of the following formula (3).

  The difference between equation (2) and equation (3) is simply a constraint term

Only increased. This constraint is an arbitrary discrete set with system parameters w ^

This means that it is accepted as a solution only if it becomes an element of. That is, the optimization problem equivalent to the above equation (2) is solved, but the solution needs to satisfy the constraints. This constraint is used to design a finite number of discrete values in order to design the parameter values to be duplicated as much as possible.

Cartesian product set of

Consists of.

As a second process, parameters having the same value are combined into one. This is a post-processing position of the construction phase. For example, assuming that w _i = w _j = w _k , that is, when the i, j, and k-th system parameters (parameter vector elements) have the same value, w _i , w _j , This corresponds to a process of deleting w _k and newly setting w _l . However, the newly added w _l is the same value as w _i and the like, and the indexes i, j, and k are equivalent to assuming that the index l is newly reassigned. In this way, even if the same value is held, it is redundant information, so that the same information can be obtained by referring to the index of the original value that points to the newly allocated index. By doing so, it is possible to obtain a set of system parameters, which are the system parameters in the same format as before, but with a reduced amount of duplicate system parameter values.

  In the third process, the system is actually operated using the obtained system parameters. In this process, the system parameters in the same format as in the conventional system are obtained in the second process, so that the above equation (1) equivalent to the conventional one can be used as the process.

  The embodiment according to the present invention mainly changes the first process. With traditional methods, constraints

Sets appearing in

Must be determined manually. In other words, the definition of this set greatly affects the system accuracy. Also, this definition varies depending on the data and the target task, and it is difficult to give an optimal definition in advance. In some cases, it may be possible to derive a definition that can build a high-accuracy model based on human prior knowledge, etc., but as a general rule, this is a very difficult and sensitive task, so it is very expensive in terms of manpower costs. It becomes a problem. In the embodiment according to the present invention, the part that determines this definition is included in the optimization problem, and is automatically determined from the data.

  Specifically, constraints

The constraints

The above equation (3) is redefined as the following equation (4).

Meaning of the expression (4), possible values as constraint 0 becomes only ± v _i, v _i is a ζ types of real-valued. For example, when ζ = 4 is designated in advance, there are nine possible values including zero for each parameter w in the model. In addition, the possible value constraint v _{i at} this time is also automatically determined from the data.

  In the following, the problem of natural language processing specific expression extraction and dependency analysis as shown in FIG. 1 will be described assuming that the embodiment of the present invention is applied. These problems are called structure prediction problems, and can be regarded as problems that receive as an input what has been converted to a graph structure or the like and output what is also represented by the graph structure.

  In the construction and execution phase described below, as shown in FIG. 2, first, the definition of the features that characterize each input and output is given manually. Here, since the feature is assumed to be defined as a function, it is equivalent to defining a set of feature extraction functions. Also, the system operation definition such as what kind of output is made when a certain input is given is given manually. This is automatically determined in accordance with the definition of the problem to be solved, here, the specific expression extraction and dependency analysis.

<System configuration of system parameter learning device>
Next, the configuration of the system parameter learning device according to the embodiment of the present invention will be described. As shown in FIG. 3, a system parameter learning device 100 according to an embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a system parameter learning processing routine described later, Can be configured with a computer including Functionally, the system parameter learning apparatus 100 includes a teacher data input unit 10, a calculation unit 20, and a system parameter storage unit 90 as shown in FIG.

  The teacher data input unit 10 receives input of teacher data. Here, as shown in FIG. 1, the teacher data is a plurality of pairs of predetermined input data and correct output data for the input data. The teacher data input unit 10 receives an input of a tuning parameter ρ used in the learning unit 40 described later. The parameter ρ is given in advance by hand.

  The calculation unit 20 includes a teacher database 30, a learning unit 40, and a duplicate parameter compression unit 60.

  Teacher data received by the teacher data input unit 10 is stored in the teacher database 30.

The learning unit 40 learns a plurality of system parameters based on teacher data stored in the teacher database 30 and a predetermined score function. Specifically, the learning unit 40 determines that each value of the plurality of system parameters is a real value v _i (i = 1,..., Ζ) with a predetermined number ζ based on the teacher data and the score function. the real value _{-v i (i = 1, ...} , ζ) satisfy the constraints in the set of discrete values consisting of 0 Prefecture, and learns a plurality of system parameters optimized. The learning unit 40 includes an initialization unit 50, a system parameter update unit 52, an auxiliary parameter update unit 54, an undetermined multiplier update unit 56, and a convergence determination unit 58.

The learning algorithm in the learning unit 40 learns system parameters w ^ = (w ₁ , w ₂ ,..., W _N ) that minimize the objective function. Specifically, the correct answer data and the supervised learning algorithm are determined manually and given as the setting of the construction phase. In the following, the objective function for learning the system parameter w will be described first.

Here, the feature extraction function f for extracting features is a function defined by a combination of input data x ^ and output data y ^. Each feature extraction function is a function of the form f _d (x ^, y ^) and is a function that returns an arbitrary real value. Here, like w ^, a set of feature extraction functions is represented by a vector notation f ^ (x ^, y ^). At this time, f _d (x ^, y ^) represents the d-th element of f ^ (x ^, y ^). Examples of features extracted by the feature extraction function are shown in FIG. 4 and FIG.

  If the number of feature extraction functions is designed to be the number N of system parameters, the score function () can be defined as a linear function as shown in the following equation (5).

That is, the system parameter w _d is the weight of the feature extraction function f _d , and if the value is large, the feature extraction function f _d more influences the selection of the output data of the system. This corresponds to weighting such that y is not selected more. In other words, the system parameter here is a value that determines the weight of the feature. The learning unit 40 learns system parameters using a supervised learning algorithm as shown in FIG.

  Although the objective function is expressed by the above equation (4), the above equation (4) is basically a discrete optimization problem because the solution has discrete constraints, and it is very difficult to obtain a solution strictly. Become a system. However, an efficient solution can be obtained by utilizing a method based on dual decomposition (eg, Non-Patent Document 3: Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein, “Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers ”, 2011, Foundations and Trends in Machine Learning). First, the constraint of the above equation (4) is decomposed based on dual decomposition.

  Here, u ^ is an auxiliary parameter. This means that the conventional optimization problem and the constraint are separated by using the equality constraint w ^ = u ^. Next, the constraint is substituted into the objective function using the extended Lagrangian relaxation of Non-Patent Document 3.

  α ^ is a Lagrange multiplier. According to the non-patent document 3, the objective function shown in the above equation (7) is obtained by the processing in the initialization unit 50, the system parameter update unit 52, the auxiliary parameter update unit 54, the undetermined multiplier update unit 56, and the convergence determination unit 58. By optimizing, in the present embodiment, system parameters with overlapping values are obtained.

  As the risk function, basically any function may be used, but here, the risk function is limited to a convex function. In the present embodiment, a hinge loss function shown in the following equation (8) is used as the risk function.

Here, E (y ^, ^* y ^) is a function indicating how much y ^ differs from ^* y ^. y ^ and ^* y ^ and as there is a large difference between ^{the, E (y ^, * y} ^) should be greater than or equal to 0 of large value, y ^ and ^* y ^ and E in the case are the same (y ^{^,} * y ^) Becomes 0.

Also, like the risk function, the regularization term is also a convex function. For example, for the regularization term, the L ₁ -norm regularization term shown in the following equation (9) is used.

  Since the risk function and the objective function are convex functions, the optimization problem of the above equation (2) is basically convex optimization, and there is always a global optimal solution.

  The initialization unit 50 sets all three types of parameters w ^, u ^, and α ^ used for optimization to zero. Since the calculation is repeated, the variable for managing the number of repetitions is t, and t = 0.

The system parameter update unit 52 includes three types of parameters w ^, u ^ ^(t) , α ^ ^(t) initialized by the initialization unit 50, or three types of parameters w updated last time in each process described later. Using {circumflex over ⁽ u ⁾ }, {circumflex over ⁽ u ⁾ } ^(t) , and {circumflex over ⁽ α ⁾ } ^(t) , the system parameter “w” ^{(t )} is updated to “w” ^{(t + 1) as described} below.

The optimal solution of w ^ when u ^ ^(t) and α ^ ^(t) are fixed at the time of the iterative calculation t is the objective function

Is the solution to the problem of finding w ^ that minimizes.

  According to the definition, when only the term related to w ^ is extracted from the above equation (10), the following equation (11) is obtained.

This optimization problem can be regarded as an optimization problem in which an L ₂ -norm regularization term is added to the above equation (2). This problem is always a convex optimization problem if the above equation (2) is a convex optimization problem, and can be easily solved using a method for solving the above equation (2).

Therefore, the system parameter update unit 52 performs the three types of parameters w ^, u ^ ^(t) , α ^ ^(t) initialized by the initialization unit 50, or the three types updated last time in each process described later. Using the parameters w ^, u ^ ^(t) and α ^ ^(t) , the system parameters w ^ ^{(t )} are updated to w ^ ^{(t + 1)} according to the above equation (11).

The auxiliary parameter update unit 54 is updated by the system parameter update unit 52 w ^ ^{(t + 1)} , the parameter α ^ ^(t) initialized by the initialization unit 50, or the parameter α ^ ^(t) updated last time. As described below, the auxiliary parameter u ^ ^{(t )} is updated to u ^ ^{(t + 1)} .

When w ^ ^{(t + 1)} and α ^ ^(t) are fixed, the optimal solution of u ^ is the objective function

Constrained to

Is the solution to the added optimization problem. First, if only terms related to u ^ are collected, an objective function expressed by the following equation (12) is obtained.

  First, considering the case of no constraints, the objective function

Is the point at which the gradient for u ^ becomes a zero vector. From the relation, the following relational expressions (13) and (14) are obtained.

Next, considering the constraints, from the above equation (12), the optimal solution u ^ = w ^ ^{(t + 1)} + α ^(t) when there is no constraint is regarded as a mapping problem to a hyperplane having ζ degrees of freedom. be able to. Therefore, as shown in FIG. 7, a process for finding a point on the hyperplane closest to u ^ = w ^ ^{(t + 1)} + α ^(t) may be performed. Actually, this processing is equivalent to one-dimensional k-means as shown in FIG. 8 (Non-Patent Document 4: Haizhou Wang and Mingzhou Song. Ckmeans.1d.dp: Optimal k-means Clustering in One Dimension by Dynamic Programming. The R Journal, 3 (2): 29.33, 2011.). Thus, each parameter can be processed independently and can be obtained very efficiently.

The undetermined multiplier updating unit 56 includes w ^ ^{(t + 1)} updated by the system parameter updating unit 52, u ^ ^{(t + 1)} updated by the auxiliary parameter updating unit 54, and the parameter α initialized by the initialization unit 50. Using La ^(t) or the parameter α ^ ^(t) updated last time, the Lagrange undetermined multiplier α ^ is updated as described below.

The direction of the optimum value when w ^ ^{(t + 1)} and u ^ ^{(t + 1)} are fixed is the objective function

Is the gradient direction with respect to α ^.

An update formula of the following formula (16) is obtained from the formula (15).

As described above, the undetermined multiplier update unit 56 is initialized by the initialization unit 50 with w ^ ^{(t + 1)} updated by the system parameter update unit 52, u ^ ^{(t + 1)} updated by the auxiliary parameter update unit 54. and parameter alpha ^{^ (t),} or by using the parameters alpha ^{^} and ^(t), which was last updated, according to the above (16), to update the Lagrange multiplier alpha ^{^ (t)} alpha ^{^} to ^{(t + 1).}

The convergence determination unit 58 determines whether or not a predetermined condition is satisfied, and repeats update processing by the system parameter update unit 52, the auxiliary parameter update unit 54, and the undetermined multiplier update unit 56 until the condition is satisfied. . Specifically, the convergence determination unit 58 determines whether or not the system parameter ＾ obtained by each of the above processes is an optimal value. More specifically, two small positive real numbers ε ₁ and ε ₂ are given, and it is determined that convergence has occurred when the following equation (17) and the following equation (18) are satisfied (see Non-Patent Document 3 above).

  If the convergence is not determined in the convergence determination by the convergence determination unit 58, t = t + 1 is returned to the system parameter update unit 52. If it is determined that it has converged, the process proceeds to the duplicate parameter compression unit 60 described later.

Based on a plurality of system parameters learned by the learning unit 40, the duplicate parameter compression unit 60 forms a group with system parameters having the same value, assigns an index number to each group, and sets the group for each group. Each of the constituent system parameters is converted into a system parameter having an index number assigned to the group, and the system parameter having the index number is stored in a system parameter storage unit 90 described later. Specifically, as defined in the above equation (5), since the score function is assumed to be in a linear format, groups are defined by system parameters that have the same values such that w _i = w _j =,. To construct.

  Assuming that K groups are created, each group is assigned an index number from 1 to K.

A set of K groups is constructed as follows. Here, when w _i = w _j = w _l ,

Is the set of system parameter original numbers,

It is defined as That is, it means that the conversion from the original system parameter number to the index number of the group having the same value as the system parameter is temporarily stored. Also, let v _{k be} the value of the k-th group, ie, w _i = w _j = w _l

Then, w _i = w _j = w _l = v _k .

The duplicate parameter compression unit 60 applies the conversion of the index number to the feature function f in the same manner. At this time, g _k is newly defined by a simple linear combination of the original feature extraction functions as in the following equation (19).

  Then, the score function of the above equation (5) is

This relationship holds. Here, the plurality of system parameters w ^ are weights of the plurality of feature extraction functions f _d (x ^, y ^) for the input data x ^ and the output data y ^. That is, the original linear function Σ ^N _{d = 1} w _d f _d (x ^, y ^) can be converted equivalently to the new linear function Σ ^K _{k = 1} v _k g _k (x ^, y ^). However, since K is the number of groups, it can be assumed that the number is much smaller than N. Moreover, since g _k is the sum of the feature functions belonging to the group, it can be easily calculated in advance.

  As described above, for each group, the duplicate parameter compression unit 60 converts each system parameter constituting the group into the system parameter of the index number assigned to the group, and converts the system parameter of the index number to In addition to saving, a score function is defined as shown in the above equation (20).

Therefore, the model size can be greatly reduced by merging overlapping portions from a set of system parameters obtained by the system parameter updating unit 52 and having many overlapping portions and compressing them into an equivalent but useless format. As an additional process, when w _d = 0, f _d contributes nothing to the output selection, so that the definition of the feature function itself can be deleted. This processing is also performed here.

The system parameter storage unit 90 stores the system parameter v _k compressed by the duplicate parameter compression unit 60 and the newly defined score function Φ (x ^, y ^; w ^).

<System configuration of information processing apparatus>
The information processing apparatus 200 performs predetermined information processing on unknown input data using the compressed system parameters obtained by the system parameter learning apparatus 100 described above. The processing results are completely the same when the system parameters are compressed and when the system parameters are not compressed. Therefore, by performing compression, it is possible to enjoy only the merit that the resources required at the time of execution can be reduced by the amount that the size of the model (the size of the system parameter itself) can be greatly reduced.

  FIG. 9 is a block diagram showing an information processing apparatus 200 according to the embodiment of the present invention. The information processing apparatus 200 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing an information processing routine to be described later, and is functionally configured as follows. Yes.

  As illustrated in FIG. 9, the information processing apparatus 200 according to the present embodiment includes an input unit 210, a system parameter storage unit 220, an information processing unit 230, and an output unit 240.

  The input unit 210 receives input data x ^.

The system parameter storage unit 220 stores a system parameter v _k compressed by the system parameter learning device 100 and a newly defined score function g _k .

The information processing unit 230 is input based on the system parameter v _k of the index number k stored in the system parameter storage unit 220 and the newly defined score function Φ (x ^, y ^; w ^). Predetermined information processing is performed on the input data x ^ received by the unit 210. Specifically, the information processing unit 230, the input data x ^ accepted by the input unit 210, the system parameter stored in the storage unit 220, and the system parameters v _k of index number k, newly defined score function Based on Φ (x ^, y ^; w ^), output data y ^ having the maximum score function Φ (x ^, y ^; w ^) is calculated using a predetermined optimization method.

The output unit 240 outputs the output data y ^ calculated by the information processing unit 230 as a result.
<Operation of system parameter learning device>
Next, the operation of the system parameter learning device 100 according to the present embodiment will be described. First, when teacher data, a parameter ρ, and a parameter ζ indicating the number of real values are input to the system parameter learning device 100, the input teacher data is input to the teacher database 30 by the system parameter learning device 100. Stored.

  Then, the system parameter learning apparatus 100 executes the system parameter learning process routine shown in FIG.

  First, in step S100, the initialization unit 50 sets all three types of optimization variables w ^, u ^, α ^ used for optimization to 0 and initializes them.

  In step S102, the initialization unit 50 initializes the variable t for managing the number of repetitions by setting t = 0.

In step S104, the three types of parameters w ^, u ^ ^(t) , α ^ ^(t) initialized in step S100 by the system parameter update unit 52, or the three types last updated in each step described later. The system parameters w ^ ^(t) to w ^ ^{(t + 1)} are updated according to the above equation (11) using the parameters w ^, u ^ ^(t) and α ^ ^(t) .

In step S106, the auxiliary parameter update unit 54 updates w ^ ^{(t + 1)} updated in step S104, the parameter α ^ ^(t) initialized in step S100, or updated last time in step S108 described later. Using the parameter α ^ ^(t) , u ^ is calculated according to the above equation (14), a point on the hyperplane closest to the calculated u ^ is found, and from the auxiliary parameter u ^ ^(t) to u ^ Update to ^{(t + 1)} .

In step S108, the undetermined multiplier updating unit 56 updates w ^ ^{(t + 1)} updated in step S104, u ^ ^{(t + 1)} updated in step S106, and the parameter α ^ initialized in step S100. ^(T) or using the parameter α ^ ^(t) updated last time in step S108, the Lagrange undetermined multiplier α ^ ^{(t )} is updated to α ^ ^{(t + 1)} according to the above equation (16).

In step S110, the convergence determination unit 58 uses the previous updated w ^ ^{(t + 1)} , the previous updated w ^ ^(t) , the updated u ^ ^{(t + 1) in} step S106, and the previous time. Based on the updated u ^ ^(t) , it is determined whether or not the convergence condition shown in the above equation (17) and the above equation (18) is satisfied. If it is determined that it has not converged, the variable t for managing repetition is incremented in step S112, the process proceeds to step S104, and the processes in steps S104 to S108 are repeated. When it determines with having converged, it transfers to step S114.

  In step S114, the duplicate parameter compression unit 60 forms a group with system parameters having the same value based on the value of the system parameter w ^ finally obtained by the update process in step S104. Each of the system parameters constituting the group is converted into a system parameter of the index number assigned to the group, the system parameter of the index number is stored in the system parameter storage unit 90, and the above equation (20) As shown in FIG. 5, a new score function is defined and stored in the system parameter storage unit 90, and the system parameter learning processing routine is terminated.

<Operation of information processing device>
Next, the operation of the information processing apparatus 200 according to this embodiment will be described. First, each of the system parameters v _k stored in the system parameter storage unit 90 of the system parameter learning device 100 and the newly defined score function Φ (x ^, y ^; w ^) Is stored in the system parameter storage unit 90. When the target input data x ^ is input to the information processing apparatus 200, the information processing apparatus 200 executes an execution processing routine shown in FIG.

  First, in step S200, the input unit 210 accepts input data x ^.

Next, in step S202, each of the system parameters v _k stored in the system parameter storage unit 220 and the newly defined score function Φ (x ^, y ^; w ^) are processed by the information processing unit 230. Read.

In step S204, the information processing unit 230, the input data x ^ accepted in step S200, respectively, and score function Φ (x ^, y ^; w ^) of the system parameters _{v k} read in step S202 and the Based on this, output data y ^ having the maximum score function Φ (x ^, y ^; w ^) is calculated using a predetermined optimization method.

  In step S206, the output unit 240 outputs the output data y ^ calculated in step S204 as a result, and the information processing routine ends.

<Experimental result>
Next, experimental results of the present embodiment are shown. In fact, in the text processing problem, since the discrete features such as words are used as so-called features that characterize the input and output at the time of identification, the total number of features is used from several million to several billions. It can often happen. FIG. 12 shows the effect of using the method described in this embodiment in the syntax analysis of natural language processing and the specific expression extraction problem.

  The horizontal axis in FIG. 12 represents the number of groups, and the vertical axis represents the analysis accuracy of the system. As can be seen from this figure, when the method of the present embodiment is used, an analysis accuracy equivalent to the conventional one can be obtained with a very small number of groups, such as 2 to 8.

  Next, it takes about 6 hours for each trial of model learning in this experimental data. The definition manually adjusted to obtain the results here is about 20 times. On the other hand, when the method of the present embodiment is used, the same result can be obtained at one time without particularly trial and error. This is a simple calculation and is equivalent to the fact that the model creation cost can be reduced to about 1/20.

As described above, the system parameter learning device according to the embodiment of the present invention accepts teacher data that is a pair of each of a plurality of input data and a plurality of correct output data for the plurality of input data. , and teacher data received, based on the score function, the set of values of each of the plurality of system parameters is the number consists of predetermined and real value v _i and a real value -v _i 0 Metropolitan discrete values By learning a plurality of system parameters that satisfy the constraints included and are optimized, an appropriate high compression model can be obtained automatically.

  Further, according to the information processing apparatus according to the embodiment of the present invention, the input data is subjected to predetermined information processing based on the compressed system parameter learned by the system parameter learning apparatus, and the output data By outputting, resources required for execution can be reduced as much as the size of the model (the size of the system parameter itself) can be greatly reduced.

  In addition, it is possible to greatly reduce the size of the system parameters themselves while maintaining almost the same accuracy as before. Specifically, in the case of a model with 100 million system parameters, since 100,000,000 × 8 bytes = 800,000,000 bytes, a capacity of 800 MB is required. However, by using this embodiment, the number of types of system parameter values can be reduced to a very small value such as about 8. In this case, 8 × 8 bytes = 64 bytes, and in terms of calculation, the compression can be performed to about 12.5 million.

  In addition, as a difference from the method shown in the prior art, in the prior art,

Set that appears in

Definition of

Is adjusted manually. At this time, since a good definition must be found by trial and error, in general, a method in which dozens of trials are executed sequentially or in parallel, and the best development set result is selected. Is used. On the other hand, in the present embodiment, manual adjustment is not necessary, and it is possible to obtain a model that maintains the same level of accuracy and compression rate as the prior art with almost one trial. . As a result, the speed and cost of model construction and selection can be reduced to a few to a few tenths as compared with the conventional case, so that it is possible to provide a practically very valuable effect.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, the system parameter storage units 90 and 220 and the teacher database 30 may be provided outside and connected to the system parameter learning device 100 and the information processing device 200 via a network.

  Moreover, although the case where the system parameter learning device 100 and the information processing device 200 are configured as separate devices has been described as an example in the above embodiment, the system parameter learning device 100 and the information processing device 200 are configured as one device. It may be configured.

  The system parameter learning device 100 and the information processing device 200 described above have a computer system inside, but if the “computer system” uses a WWW system, a homepage providing environment (or display environment) ).

  For example, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program may be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Teacher data input part 20 Operation part 30 Teacher database 40 Learning part 50 Initialization part 52 System parameter update part 54 Auxiliary parameter update part 56 Undecided multiplier update part 58 Convergence determination part 60 Duplicate parameter compression part 90 System parameter storage part 100 System parameter Learning device 200 Information processing device 210 Input unit 220 System parameter storage unit 230 Information processing unit 240 Output unit

Claims (8)

Learning a plurality of system parameters set in an information processing system that performs predetermined information processing on input data and outputs the output data using a score function for determining the score of the output data with respect to the input data A system parameter learning device,
A teacher data input unit that receives teacher data that is a pair of each of a plurality of input data and each of a plurality of correct output data for the plurality of input data;
Based on the teacher data received by the teacher data input unit and the score function, each value of the plurality of system parameters is a predetermined number of real values v _i (i = 1,..., Ζ). the real value _{-v i (i = 1, ···} , ζ) and satisfies the constraints in the set of discrete values of zeros Prefecture, and a learning unit that learns a plurality of system parameters that are optimized A system parameter learning device. The system parameter learning device according to claim 1, wherein the learning unit learns the optimized N system parameters w ^ according to the following expression (1).
However,
Is the teacher data
The output data obtained when the predetermined information processing is performed on the input data x _i using the system parameter w ^ is the teacher data
Is a risk function that returns a value indicating how wrong the correct output data y _i is, and Ω (w ^) is a regularization term,
Is a Cartesian product set of the discrete values for each of the N system parameters. 3. The system parameter learning device according to claim 2, wherein the learning unit learns the optimized N system parameters w ^ according to the following equation (2).
Here, α ^ is a Lagrange undetermined multiplier, ρ is a predetermined tuning parameter, and u ^ is an auxiliary parameter.   Based on the plurality of system parameters learned by the learning unit, configure a group with system parameters having the same value, assign an index number to each group, and for each group, a system parameter that configures the group The system parameter learning according to claim 1, further comprising a duplicate parameter compression unit that converts each into a system parameter of the index number assigned to the group and stores the system parameter of the index number. apparatus. The plurality of system parameters are weights of a plurality of feature extraction functions f _d (x ^, y ^) for input data x ^ and output data y ^,
The duplicate parameter compression unit converts, for each group, each system parameter constituting the group into a system parameter of the index number assigned to the group, stores the system parameter of the index number, and The system parameter learning apparatus according to claim 4, wherein the score function is defined and stored as shown in equation (3).
However, Φ (x ^, y ^ ; w ^) is a score function, v _k is a system parameter of the index number k, which is granted to the group. An input unit for receiving input data;
Based on the score function and the system parameter of the index number of each group stored by the system parameter learning device according to claim 4 or 5, the predetermined information for the input data received in the input unit An information processing unit that performs processing and outputs output data;
Including an information processing apparatus. An information processing system that includes a teacher data input unit and a learning unit, performs a predetermined information process on input data using a score function for determining a score of the output data with respect to the input data, and outputs the output data A system parameter learning method in a system parameter learning device for learning a plurality of system parameters set in
The teacher data input unit accepts teacher data that is a pair of each of a plurality of input data and a plurality of correct output data for the plurality of input data;
Based on the teacher data received by the learning data input unit and the score function, the learning unit sets each of the plurality of system parameters to a predetermined number of real values v _i (i = 1,. .., Ζ) and real values −v _i (i = 1,..., Ζ) and 0 satisfy the constraints included in the set of discrete values and optimize the plurality of system parameters. Learning System parameter learning method.   The program for functioning a computer as each part which comprises the system parameter learning apparatus of any one of Claims 1-5, or the information processing apparatus of Claim 6.