JP6831307B2 - Solution calculation device, solution calculation method and solution calculation program - Google Patents

Description

The present invention relates to a solution calculation device, a solution calculation method, and a solution calculation program.

It is known that machine learning problems are generally formulated as loss function minimization problems. For example, the regression problem is formulated as a minimization problem such as a square loss function or a Huber loss function. Similarly, for example, the classification problem is formulated as a minimization problem such as a logistic loss function or a hinge loss function.

On the other hand, submodular structure sparse constraints are used to improve the generalization performance of machine learning and the interpretability of results. By using the submodular structure sparse constraint, for example, learning can be performed while considering the hierarchical structure of parameters and the structure of groups.

By combining these to make a minimization problem under the submodular structure sparse constraint, various machine learning problems can be formulated. Various methods have been proposed for solving such machine learning problems (Non-Patent Documents 1 to 3).

F. Bach. Structured sparsity-inducing norms through submodular functions. In Advances in Neural Information Processing Systems, pages 118-126, 2010. M. El Halabi, L. Baldassarre, and V. Cevher. To convexify or not? Regression with clustering penalties on graphs. In Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2013 IEEE 5th International Workshop on, pages 21-24 .IEEE, 2013. R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde. Model-based compressive sensing. IEEE Transactions on Information Theory, 56 (4): 1982-2001, 2010.

However, the above method has some problems. For example, the methods disclosed in Non-Patent Documents 1 and 2 have a problem that the accuracy of the obtained solution is low. Further, for example, the method disclosed in Non-Patent Document 3 has a problem that only a squared loss function can be handled as a function to be minimized and a long time is required to obtain a solution.

The present invention has been made in view of the above problems, and an object of the present invention is to obtain a solution of a minimization problem under a submodular structure sparse constraint using a general loss function at high speed and with high accuracy.

In order to solve the above problem, it is a solution calculation device that calculates the solution of the minimization problem under the submodular structure sparse constraint using the loss function, and the loss function, the first submodular function, and the constraint constant. An input unit for inputting, an initialization unit that initializes the number of repetitions t and the solution of the minimization problem, and a gradient descent calculation unit that calculates a post-descent vector obtained by gradient descent of the solution at the t-th time. , A set that minimizes the second submodular function determined from the post-descent vector calculated by the gradient descent calculation unit and the first submodular function, and the value of the first submodular function is the value. Truncation calculation or exact minimization calculation based on the submodular function minimization unit that calculates a set that is less than or equal to the constraint constant, the solution at the t-th time, and the set calculated by the submodular function minimization unit. The solution calculation unit for calculating the solution at the t + 1th time and the end condition determination unit for determining whether or not the predetermined end condition is satisfied are provided, and the end condition determination unit has the predetermined end condition. When it is determined that the above is not satisfied, 1 is added to the number of repetitions t, and the gradient descent calculation unit calculates the post-descent vector.

A solution of a minimization problem under a submodular structure sparse constraint using a general loss function can be obtained at high speed and with high accuracy.

It is a figure which shows an example of the structure of the solution calculation apparatus in embodiment of this invention. It is a figure which shows an example of the hardware composition of the solution calculation apparatus in embodiment of this invention. It is a figure which shows an example of the functional structure of the solution calculation apparatus in embodiment of this invention. It is a flowchart which shows an example of the whole process executed by the solution calculation apparatus in embodiment of this invention. It is a figure which shows an example of a correct answer signal. It is a figure (the 1) which shows the comparative example of this invention and the prior art. It is a figure (the 2) which shows the comparative example of this invention and the prior art.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Configuration of solution calculation device 10>
First, the configuration of the solution calculation device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram showing an example of the configuration of the solution calculation device 10 according to the embodiment of the present invention.

The solution calculation device 10 shown in FIG. 1 is a computer that calculates a solution of a minimization problem under a submodular structure sparse constraint using a general loss function. The solution calculation program 100 is installed in the solution calculation device 10 shown in FIG. The solution calculation program 100 may be a program group composed of a plurality of modules.

The solution calculation device 10 according to the embodiment of the present invention has a loss function f: R ^d → R, a submodular function g: 2 ^{{1, ..., D}} → R, and a constraint constant by the solution calculation program 100. Input c and output the solution vector w of the minimization problem. At this time, the output solution vector w satisfies the submodular structure sparse constraint g (supp (w)) ≦ c and minimizes the value f (w) of the loss function.

Note that R and R ^d are a one-dimensional real number space and a d-dimensional real number space, respectively. 2 ^{{1, ..., d}} is a power set of sets {1, ..., d}. The solution vector w is a d-dimensional real vector. Hereinafter, the solution vector is also simply referred to as a “solution”.

More specifically, the solution calculation device 10 according to the embodiment of the present invention repeats the following (1) to (3) while increasing the variable t indicating the number of iterations by 1 until a predetermined end condition is satisfied. By executing, the solution wt ^{+ 1} is calculated. Then, the solution calculation device 10 according to the embodiment of the present invention outputs the solution w = w ^{t + 1} calculated by the solution calculation program 100.

(1) in the gradient direction of the loss function f to calculate the descent after vector v moves the current solution w ^t (gradient descent).

(2) A set S that minimizes the submodular function G determined by the post-descent vector v and the submodular function g is calculated.

(3) and current solution w ^t, based on a set S calculated in the above (2), by truncation calculation or strict minimization calculations to calculate the next solution w ^{t + 1.}

The configuration of the solution calculation device 10 shown in FIG. 1 is an example, and may be another configuration. For example, the solution calculation device 10 shown in FIG. 1 may be composed of a plurality of computers.

<Hardware configuration of solution calculation device 10>
Next, the hardware configuration of the solution calculation device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 2 is a diagram showing an example of the hardware configuration of the solution calculation device 10 according to the embodiment of the present invention.

The solution calculation device 10 shown in FIG. 2 includes an input device 11, a display device 12, an external I / F 13, a RAM (Random Access Memory) 14, a ROM (Read Only Memory) 15, and a CPU (Central Processing Unit). It has 16, a communication I / F 17, and an auxiliary storage device 18. Each of these hardware is connected so as to be able to communicate with each other via the bus B.

The input device 11 is, for example, a keyboard, a mouse, a touch panel, or the like, and is used for a user to input various operations. The display device 12 is, for example, a display or the like, and displays the processing result of the solution calculation device 10.

The external I / F 13 is an interface with an external device. The external device includes a recording medium 13a and the like. The solution calculation device 10 can read and write the recording medium 13a and the like via the external I / F 13. The solution calculation program 100 or the like may be recorded on the recording medium 13a.

The recording medium 13a includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), a USB (Universal Serial Bus) memory card, and the like.

The RAM 14 is a volatile semiconductor memory that temporarily holds programs and data. The ROM 15 is a non-volatile semiconductor memory capable of holding programs and data even when the power is turned off. The ROM 15 stores, for example, OS (Operating System) settings, network settings, and the like.

The CPU 16 is an arithmetic unit that reads a program or data from the ROM 15 or the auxiliary storage device 18 or the like onto the RAM 14 and executes processing.

The communication I / F 17 is an interface for connecting the solution calculation device 10 to the network. The solution calculation program 100 may be acquired (downloaded) from a predetermined server or the like via the communication I / F17.

The auxiliary storage device 18 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and is a non-volatile storage device that stores programs and data. The programs and data stored in the auxiliary storage device 18 include, for example, an OS, an application program that realizes various functions on the OS, a solution calculation program 100, and the like.

The solution calculation device 10 according to the embodiment of the present invention can realize various processes described later by having the hardware configuration shown in FIG.

<Functional configuration of solution calculation device 10>
Next, the functional configuration of the solution calculation device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a diagram showing an example of the functional configuration of the solution calculation device 10 according to the embodiment of the present invention.

The solution calculation device 10 shown in FIG. 3 includes an input unit 101, an initialization unit 102, a gradient descent calculation unit 103, a submodular function minimization unit 104, a solution calculation unit 105, and an end condition determination unit 106. It has an output unit 107. Each of these parts is realized by a process in which the solution calculation program 100 causes the CPU 16 to execute.

The input unit 101 inputs a minimization problem (that is, a loss function f, a submodular function g, and a constraint constant c). Note that these information may be input by the user, for example, or may be input from another program or the like. Alternatively, it may be input by reading an electronic file stored in the auxiliary storage device 18 or the like.

The initialization unit 102 initializes the variable t indicating the number of iterations to “0” and initializes the current solution w ⁰ to a zero vector.

Gradient descent calculation unit 103 calculates the drop after vector v obtained by gradient descent the current solution w ^t.

The submodular function minimization unit 104 calculates a set S that minimizes the submodular function G determined from the post-descent vector v calculated by the gradient descent calculation unit 103 and the submodular function g. Here, the set S is a subset of the set {1, ..., D} based on natural numbers from 1 to d.

Solution calculating section 105, a current solution w ^t, based on the set S submodular function minimization unit 104 calculates, by truncating calculation or strict minimization calculations to calculate the next solution w ^{t + 1.} The solution calculation unit 105 includes a truncation calculation unit 111 that calculates the next solution wt ^{+ 1} by the truncation calculation, and a strict minimization calculation unit 112 that calculates the next solution w ^{t + 1} by the exact minimization calculation.

The end condition determination unit 106 determines whether or not a predetermined end condition is satisfied. Further, when the end condition determination unit 106 determines that the predetermined end condition is not satisfied, "1" is added to the variable t indicating the number of repetitions.

When the end condition determination unit 106 determines that the predetermined end condition is satisfied, the output unit 107 outputs the solution w = w ^{t + 1} calculated by the solution calculation unit 105. The output destination by the output unit 107 may be, for example, the display device 12 or another program. Alternatively, it may be another device or the like connected to the solution calculation device 10 via a network or the like.

<Details of processing>
Next, the details of the processing of the solution calculation device 10 in the embodiment of the present invention will be described. Hereinafter, the overall processing executed by the solution calculation device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flowchart showing an example of the overall processing executed by the solution calculation device 10 according to the embodiment of the present invention.

First, the input unit 101 inputs the loss function f, the submodular function g, and the constraint constant c (step S101).

Next, the initialization unit 102 initializes the variable t indicating the number of iterations to “0” and initializes the current solution w ⁰ to a zero vector (step S102).

Then, gradient descent calculation unit 103 calculates the drop after vector v obtained by gradient descent the current solution ^{w t} (step S103). That is, the gradient descent calculation unit 103 calculates the post-descent vector v by the following equation 1.

Here, η _t is a step size parameter. The step size parameters can be determined by grid search, backtracking, or the like.

Next, the submodular function minimization unit 104 calculates a set S that minimizes the submodular function G determined from the post-descent vector v calculated by the gradient descent calculation unit 103 and the submodular function g (step S104). That is, the submodular function minimization unit 104 calculates the set S by the following equation 2.

Here, the submodular function G is defined by the following equation 3. Further, α _t is a regularization parameter, and is selected sufficiently small so that the obtained set S satisfies g (S) ≦ c. This can be achieved by using parametric submodular minimization, backtracking, or the like.

Next, solution calculating section 105, a current solution w ^t, based on the set S submodular function minimization unit 104 calculates, by truncating calculation or strict minimization calculations to calculate the next solution w ^{t + 1} ( Step S105).

That is, when calculating the next solution ^{w t + 1} by truncation calculations, solution calculating section 105, the truncated calculation unit 111, by Equation 4 below to calculate the next solution ^{w t + 1.}

On the other hand, when calculating the next solution w ^{t + 1} by rigorous minimization calculations, solution calculating section 105, the exact minimization calculation unit 112, by Equation 5 below to calculate the next solution w ^{t + 1.}

It should be noted that it may be arbitrarily determined whether to calculate the next solution wt ^{+ 1} by using the truncation calculation or the strict minimization calculation.

Next, the end condition determination unit 106 determines whether or not the predetermined end condition is satisfied (step S106). The end condition determination unit 106 may use, for example, any of the following (a) to (d) as a predetermined end condition.

(A) The amount of reduction in training error f (w ^t ) -f (w ^{t + 1} ) is smaller than the predetermined threshold value (b) The amount of reduction in validation error is smaller than the predetermined threshold value (c) Solution w ⁰ , The minimum value of the validation error calculated from w ¹ , ..., W ^{t + 1} is not updated during the predetermined number of iterations. (D) The value of t is larger than the predetermined value. In step S106, When it is determined that the predetermined end condition is not satisfied, the end condition determination unit 106 adds "1" to the variable t indicating the number of repetitions (step S107). Then, the process returns to the process of step S103 described above. As a result, the above steps S103 to S107 are repeatedly executed until a predetermined end condition is satisfied.

On the other hand, if it is determined in step S106 that the predetermined end condition is satisfied, the output unit 107 outputs the solution w = w ^{t + 1} calculated by the solution calculation unit 105 (step S108).

As described above, the solution calculation device 10 according to the embodiment of the present invention can calculate the solution w of the minimization problem under the submodular structure sparse constraint using the general loss function f.

<Comparison example>
Next, a comparative example between the present invention and the prior art in the case where two experiments using artificial data are performed will be described. As the prior art, FISTA (see Non-Patent Document 1), Majorization-Minimation (hereinafter referred to as “MM”; see Non-Patent Document 2), and Model-Based CoSaMP (hereinafter referred to as “MCoSaMP”). (See Reference 3) and.

In the first experiment, we solved the signal restoration problem using the Haar wavelet basis. As the correct answer signal, the signal shown in FIG. 5 was used. Further, the loss function f is defined as the squared loss function, and the submodular function g is defined to induce the hierarchical structure sparseness of the wavelet basis. FIG. 6 shows a comparative example of the test errors of the solutions obtained by the present invention and the prior art in this case. FIG. 6 is a diagram (No. 1) showing a comparative example between the present invention and the prior art.

In the comparative example shown in FIG. 6, the horizontal axis represents the execution time and the vertical axis represents the test error. A small test error indicates that the solution is highly accurate. As shown in FIG. 6, it can be seen that in the present invention, a solution is obtained at high speed and with high accuracy as compared with the prior art, regardless of whether the truncation calculation or the strict minimization calculation is used.

Next, in the second experiment, we solved the multiclass classification problem. The true class parameters were randomly generated to be the sum of the group sparse vector and the sparse vector. The loss function f is a logistic function, and the submodular function g is defined to induce the robust group sparseness of the class parameters. FIG. 7 shows a comparative example of the test errors of the solutions obtained by the present invention and the prior art in this case. FIG. 7 is a diagram (No. 2) showing a comparative example between the present invention and the prior art.

In the comparative example shown in FIG. 7, the horizontal axis is the execution time and the vertical axis is the test error, as in FIG. As shown in FIG. 7, it can be seen that in the present invention, a solution is obtained at high speed and with high accuracy as compared with the prior art. Since MCoSaMP cannot be applied to the logistic loss function, it is not shown in the comparative example shown in FIG. 7.

As described above, according to the solution calculation device 10 according to the embodiment of the present invention, the solution w of the minimization problem under the submodular structure sparse constraint using the general loss function f is compared with the prior art. It can be calculated at high speed and with high accuracy.

The present invention is not limited to the above-described embodiment disclosed specifically, and various modifications and modifications can be made without departing from the scope of claims.

10 Solution calculation device 100 Solution calculation program 101 Input unit 102 Initialization unit 103 Gradient descent calculation unit 104 Submodular function minimization unit 105 Solution calculation unit 106 End condition judgment unit 107 Output unit 111 Truncation calculation unit 112 Strict minimization calculation unit

Claims (5)

It is a solution calculation device that calculates the solution of the minimization problem under the submodular structure sparse constraint using the loss function.
Input the loss function f: R ^d → R of the minimization problem, the first submodular function g: 2 ^{{1, ..., D}} → R representing the submodular structure sparse constraint, and the constraint constant c. Input section and
An initialization unit that initializes the number of iterations t and the solution of the minimization problem,
A gradient descent calculation unit that calculates a post-descent vector v in which the solution at the t-th time is lowered in the gradient direction of the loss function f ,
It is a set S ⊆ {1, ..., D} that minimizes the second submodular function G determined from the post-descent vector v calculated by the gradient descent calculation unit and the first submodular function g. A submodular function minimization unit that calculates a set S such that the value of the first submodular function g is equal to or less than the constraint constant c .
A solution calculation unit that calculates the solution at the t + 1th time by truncation calculation or strict minimization calculation based on the solution at the t-th time and the set S calculated by the submodular function minimization unit.
An end condition determination unit that determines whether or not a predetermined end condition is satisfied, and an end condition determination unit
Have,
The second submodular function G is
Wherein the elements of the drop after the vector v _{v i} (however, i = 1, ···, d ), the alpha _t as the regularization parameter, any set A⊆ {1, ···, d} with respect to G (A) = g (A) -α _t × ( sum of squares of vi with _i ∈ A) ,
The end condition determination unit
A solution calculation device, characterized in that, when it is determined that the predetermined end condition is not satisfied, 1 is added to the number of repetitions t, and the gradient descent calculation unit calculates the post-descent vector v . The first aspect of claim 1, wherein when the end condition determination unit determines that the predetermined end condition is satisfied, the solution calculation unit has an output unit that outputs the solution at the t + 1th time calculated by the solution calculation unit. Solution calculation device. The end condition is
The amount of reduction in training error is smaller than the predetermined threshold value, the amount of reduction in validation error is smaller than the predetermined threshold value, the minimum value of validation error is not updated during the predetermined number of repetitions, or the number of repetitions. The solution calculation device according to claim 1 or 2, wherein the value of t is either larger than a predetermined value. A computer that calculates the solution of a minimization problem under a submodular structure sparse constraint using a loss function
Input the loss function f: R ^d → R of the minimization problem, the first submodular function g: 2 ^{{1, ..., D}} → R representing the submodular structure sparse constraint, and the constraint constant c. Input procedure to be done and
An initialization procedure that initializes the number of iterations t and the solution of the minimization problem, and
A gradient descent calculation procedure for calculating a post-descent vector v in which the solution at the t-th time is lowered in the gradient direction of the loss function f, and
A set S ⊆ {1, ..., D} that minimizes the second submodular function G determined by the post-descent vector v calculated by the gradient descent calculation procedure and the first submodular function g. Then, the procedure for minimizing the submodular function for calculating the set S such that the value of the first submodular function g is equal to or less than the constraint constant c , and
A solution calculation procedure for calculating the solution at the t + 1th time by a truncation calculation or an exact minimization calculation based on the solution at the t-th time and the set S calculated by the submodular function minimization procedure.
The end condition judgment procedure for determining whether or not the predetermined end condition is satisfied, and the end condition judgment procedure
And
The second submodular function G is
Wherein the elements of the drop after the vector v _{v i} (however, i = 1, ···, d ), the alpha _t as the regularization parameter, any set A⊆ {1, ···, d} with respect to G (A) = g (A) -α _t × ( sum of squares of vi with _i ∈ A) ,
The end condition determination procedure is
A solution calculation method characterized in that when it is determined that the predetermined end condition is not satisfied, 1 is added to the number of repetitions t to calculate the post-descent vector v by the gradient descent calculation procedure. A computer that calculates the solution of a minimization problem under a submodular structure sparse constraint using a loss function.
Input the loss function f: R ^d → R of the minimization problem, the first submodular function g: 2 ^{{1, ..., D}} → R representing the submodular structure sparse constraint, and the constraint constant c. Input procedure to be done and
An initialization procedure that initializes the number of iterations t and the solution of the minimization problem, and
A gradient descent calculation procedure for calculating a post-descent vector v in which the solution at the t-th time is lowered in the gradient direction of the loss function f, and
A set S ⊆ {1, ..., D} that minimizes the second submodular function G determined by the post-descent vector v calculated by the gradient descent calculation procedure and the first submodular function g. Then, the procedure for minimizing the submodular function for calculating the set S such that the value of the first submodular function g is equal to or less than the constraint constant c , and
A solution calculation procedure for calculating the solution at the t + 1th time by a truncation calculation or an exact minimization calculation based on the solution at the t-th time and the set S calculated by the submodular function minimization procedure.
The end condition judgment procedure for determining whether or not the predetermined end condition is satisfied, and the end condition judgment procedure
To execute,
The second submodular function G is
Wherein the elements of the drop after the vector v _{v i} (however, i = 1, ···, d ), the alpha _t as the regularization parameter, any set A⊆ {1, ···, d} with respect to G (A) = g (A) -α _t × ( sum of squares of vi with _i ∈ A) ,
The end condition determination procedure is
A solution calculation program characterized in that when it is determined that the predetermined end condition is not satisfied, 1 is added to the number of repetitions t to calculate the post-descent vector v by the gradient descent calculation procedure.